2023-10-20T03:19:58.771 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-20T03:19:58.772 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi --description rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} --verbose --archive /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476 -- /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/orig.config.yaml 2023-10-20T03:19:58.813 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-20T03:19:58.897 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476 branch: wip-yuri10-testing-2023-10-11-0812 description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: null first_in_suite: false job_id: '7432476' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri10-testing-2023-10-11-0812 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_async_recovery_min_cost: 1 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from sha1: 8a5df55d22d3638066184c241442ff00fdc0ebdd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8a5df55d22d3638066184c241442ff00fdc0ebdd thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri10-testing-2023-10-11-0812 sha1: 8a5df55d22d3638066184c241442ff00fdc0ebdd owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 9009 sha1: 8a5df55d22d3638066184c241442ff00fdc0ebdd sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2023-10-11-0812 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8a5df55d22d3638066184c241442ff00fdc0ebdd targets: smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKVOLgduYY8Px3G6GDW8+lkODtfqJDRiQdtvy3CJdcBka0VxAiKn06fsijKrvDtLUnWqN7FSFQQxQL+f/uE7L0A= smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPl2EW4umy7LmnRQKJV/eZwxZaWNyUCOhFdnsV1xnxguPQ3cGwHK2gDCmNjhoifsNP69ssqA/e2XaUh7m5UejZo= smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEthpyLSta/xLwOp5moRS9tdSUTC7b2/1Ub3fauZiI4GC4QFVwvwI9RehZegOcyuYvXLyZr+5XHJP/D2yxtj0Lk= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCiptNh76uIuJqcjiE0raXNpFMJh2P8PY0fynAbDKQwiRO6c+CQE0wHU+LuDdd/46qgoWu9/nM+WCTFcmVQmPvQ= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-19_16:23:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-20T03:19:58.898 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa; will attempt to use it 2023-10-20T03:19:58.899 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks 2023-10-20T03:19:58.899 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-20T03:19:58.901 INFO:teuthology.task.internal:Checking packages... 2023-10-20T03:19:58.923 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8a5df55d22d3638066184c241442ff00fdc0ebdd' 2023-10-20T03:19:58.924 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-20T03:19:58.924 INFO:teuthology.packaging:ref: None 2023-10-20T03:19:58.924 INFO:teuthology.packaging:tag: None 2023-10-20T03:19:58.925 INFO:teuthology.packaging:branch: wip-yuri10-testing-2023-10-11-0812 2023-10-20T03:19:58.925 INFO:teuthology.packaging:sha1: 8a5df55d22d3638066184c241442ff00fdc0ebdd 2023-10-20T03:19:58.925 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri10-testing-2023-10-11-0812 2023-10-20T03:19:59.162 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:19:59.163 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-20T03:19:59.171 INFO:teuthology.task.internal:no buildpackages task found 2023-10-20T03:19:59.171 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-20T03:19:59.190 INFO:teuthology.task.internal:Saving configuration 2023-10-20T03:19:59.206 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-20T03:19:59.264 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-20T03:19:59.290 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-20 03:14:22.762626', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF+mbuiTkb3+MhAlwP1Kx/ZpZPpKfVJ7j0598ofM+efe'} 2023-10-20T03:19:59.314 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-20 03:14:22.766373', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILD15hahFwZFLp0HdGFV0ZFrEazEuLdC5F8pFFj13ZYH'} 2023-10-20T03:19:59.336 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-20 03:14:22.764678', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF3LddGaluBktlOX8OST7Q1J90Q0LooRG/g42AHjSYV7'} 2023-10-20T03:19:59.359 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-20 03:14:22.768050', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEPDgZeKj1re25jeMLgqKs/X/sTqrazKwU1RK59+mw6t'} 2023-10-20T03:19:59.359 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-20T03:19:59.365 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2023-10-20T03:19:59.366 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2023-10-20T03:19:59.366 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2023-10-20T03:19:59.366 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2023-10-20T03:19:59.366 INFO:teuthology.run_tasks:Running task console_log... 2023-10-20T03:19:59.502 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb8382b3a60>, signals=[15]) 2023-10-20T03:19:59.503 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-20T03:19:59.509 INFO:teuthology.task.internal:Opening connections... 2023-10-20T03:19:59.510 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2023-10-20T03:19:59.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:19:59.590 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2023-10-20T03:19:59.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:19:59.667 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2023-10-20T03:19:59.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:19:59.746 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2023-10-20T03:19:59.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:19:59.825 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-20T03:19:59.834 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2023-10-20T03:19:59.841 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2023-10-20T03:19:59.841 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2023-10-20T03:19:59.888 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-20T03:19:59.888 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="Ubuntu" 2023-10-20T03:19:59.888 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="22.04" 2023-10-20T03:19:59.889 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-20T03:19:59.889 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_CODENAME=jammy 2023-10-20T03:19:59.889 INFO:teuthology.orchestra.run.smithi033.stdout:ID=ubuntu 2023-10-20T03:19:59.889 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE=debian 2023-10-20T03:19:59.889 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-20T03:19:59.890 INFO:teuthology.orchestra.run.smithi033.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-20T03:19:59.890 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-20T03:19:59.890 INFO:teuthology.orchestra.run.smithi033.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-20T03:19:59.890 INFO:teuthology.orchestra.run.smithi033.stdout:UBUNTU_CODENAME=jammy 2023-10-20T03:19:59.891 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2023-10-20T03:19:59.915 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2023-10-20T03:19:59.921 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2023-10-20T03:19:59.922 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2023-10-20T03:19:59.970 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-20T03:19:59.970 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="Ubuntu" 2023-10-20T03:19:59.970 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="22.04" 2023-10-20T03:19:59.971 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-20T03:19:59.971 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_CODENAME=jammy 2023-10-20T03:19:59.971 INFO:teuthology.orchestra.run.smithi040.stdout:ID=ubuntu 2023-10-20T03:19:59.971 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE=debian 2023-10-20T03:19:59.972 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-20T03:19:59.972 INFO:teuthology.orchestra.run.smithi040.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-20T03:19:59.972 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-20T03:19:59.972 INFO:teuthology.orchestra.run.smithi040.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-20T03:19:59.972 INFO:teuthology.orchestra.run.smithi040.stdout:UBUNTU_CODENAME=jammy 2023-10-20T03:19:59.973 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2023-10-20T03:19:59.998 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2023-10-20T03:20:00.004 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2023-10-20T03:20:00.004 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2023-10-20T03:20:00.052 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-20T03:20:00.053 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="Ubuntu" 2023-10-20T03:20:00.053 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="22.04" 2023-10-20T03:20:00.053 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-20T03:20:00.053 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_CODENAME=jammy 2023-10-20T03:20:00.054 INFO:teuthology.orchestra.run.smithi064.stdout:ID=ubuntu 2023-10-20T03:20:00.054 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE=debian 2023-10-20T03:20:00.054 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-20T03:20:00.054 INFO:teuthology.orchestra.run.smithi064.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-20T03:20:00.054 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-20T03:20:00.055 INFO:teuthology.orchestra.run.smithi064.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-20T03:20:00.055 INFO:teuthology.orchestra.run.smithi064.stdout:UBUNTU_CODENAME=jammy 2023-10-20T03:20:00.055 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2023-10-20T03:20:00.081 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2023-10-20T03:20:00.087 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2023-10-20T03:20:00.088 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2023-10-20T03:20:00.136 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-20T03:20:00.136 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="Ubuntu" 2023-10-20T03:20:00.137 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="22.04" 2023-10-20T03:20:00.137 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-20T03:20:00.137 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_CODENAME=jammy 2023-10-20T03:20:00.137 INFO:teuthology.orchestra.run.smithi170.stdout:ID=ubuntu 2023-10-20T03:20:00.138 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE=debian 2023-10-20T03:20:00.138 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-20T03:20:00.138 INFO:teuthology.orchestra.run.smithi170.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-20T03:20:00.138 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-20T03:20:00.139 INFO:teuthology.orchestra.run.smithi170.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-20T03:20:00.139 INFO:teuthology.orchestra.run.smithi170.stdout:UBUNTU_CODENAME=jammy 2023-10-20T03:20:00.139 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2023-10-20T03:20:00.165 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-20T03:20:00.177 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-20T03:20:00.189 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-20T03:20:00.189 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2023-10-20T03:20:00.192 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2023-10-20T03:20:00.195 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2023-10-20T03:20:00.197 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2023-10-20T03:20:00.202 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-20T03:20:00.213 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-20T03:20:00.213 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2023-10-20T03:20:00.237 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2023-10-20T03:20:00.243 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2023-10-20T03:20:00.245 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2023-10-20T03:20:00.264 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-20T03:20:00.359 INFO:teuthology.run_tasks:Running task kernel... 2023-10-20T03:20:00.375 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-20T03:20:00.375 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-20T03:20:00.376 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-20T03:20:00.377 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-20T03:20:00.377 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-20T03:20:00.378 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-20T03:20:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:20:00.383 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2023-10-20T03:20:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:20:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:20:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:20:00.386 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-10-20T03:20:00.386 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2023-10-20T03:20:00.387 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2023-10-20T03:20:00.430 INFO:teuthology.orchestra.run.smithi040.stdout:5.15.0-56-generic 2023-10-20T03:20:00.431 INFO:teuthology.task.kernel:Running kernel on smithi040: 5.15.0-56-generic 2023-10-20T03:20:00.431 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get clean 2023-10-20T03:20:00.433 INFO:teuthology.orchestra.run.smithi170.stdout:5.15.0-56-generic 2023-10-20T03:20:00.433 INFO:teuthology.orchestra.run.smithi033.stdout:5.15.0-56-generic 2023-10-20T03:20:00.433 INFO:teuthology.task.kernel:Running kernel on smithi170: 5.15.0-56-generic 2023-10-20T03:20:00.433 DEBUG:teuthology.orchestra.run.smithi170:> sudo apt-get clean 2023-10-20T03:20:00.434 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.15.0-56-generic 2023-10-20T03:20:00.434 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get clean 2023-10-20T03:20:00.435 INFO:teuthology.orchestra.run.smithi064.stdout:5.15.0-56-generic 2023-10-20T03:20:00.436 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.15.0-56-generic 2023-10-20T03:20:00.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get clean 2023-10-20T03:20:00.552 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get update 2023-10-20T03:20:00.563 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2023-10-20T03:20:00.576 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2023-10-20T03:20:00.580 DEBUG:teuthology.orchestra.run.smithi170:> sudo apt-get update 2023-10-20T03:20:00.699 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-20T03:20:00.725 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-20T03:20:00.735 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T03:20:00.735 INFO:teuthology.orchestra.run.smithi170.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-20T03:20:00.735 INFO:teuthology.orchestra.run.smithi170.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-20T03:20:00.796 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T03:20:00.814 INFO:teuthology.orchestra.run.smithi170.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-20T03:20:00.843 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T03:20:00.855 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T03:20:00.865 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-20T03:20:00.881 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-20T03:20:00.898 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [896 kB] 2023-10-20T03:20:00.912 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-20T03:20:00.925 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-20T03:20:00.947 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [896 kB] 2023-10-20T03:20:00.980 INFO:teuthology.orchestra.run.smithi170.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [353 kB] 2023-10-20T03:20:00.989 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [353 kB] 2023-10-20T03:20:00.994 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [180 kB] 2023-10-20T03:20:00.996 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-20T03:20:00.996 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,016 kB] 2023-10-20T03:20:01.013 INFO:teuthology.orchestra.run.smithi040.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-20T03:20:01.014 INFO:teuthology.orchestra.run.smithi040.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [164 kB] 2023-10-20T03:20:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-10-20T03:20:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-10-20T03:20:01.023 INFO:teuthology.orchestra.run.smithi040.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [793 kB] 2023-10-20T03:20:01.030 INFO:teuthology.orchestra.run.smithi040.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-20T03:20:01.031 INFO:teuthology.orchestra.run.smithi040.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-10-20T03:20:01.032 INFO:teuthology.orchestra.run.smithi040.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-20T03:20:01.032 INFO:teuthology.orchestra.run.smithi040.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-20T03:20:01.033 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [353 kB] 2023-10-20T03:20:01.038 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [180 kB] 2023-10-20T03:20:01.041 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-20T03:20:01.042 INFO:teuthology.orchestra.run.smithi170.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [896 kB] 2023-10-20T03:20:01.043 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,016 kB] 2023-10-20T03:20:01.057 INFO:teuthology.orchestra.run.smithi170.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,103 kB] 2023-10-20T03:20:01.058 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-20T03:20:01.059 INFO:teuthology.orchestra.run.smithi033.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [164 kB] 2023-10-20T03:20:01.061 INFO:teuthology.orchestra.run.smithi033.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-10-20T03:20:01.062 INFO:teuthology.orchestra.run.smithi033.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [793 kB] 2023-10-20T03:20:01.071 INFO:teuthology.orchestra.run.smithi170.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [180 kB] 2023-10-20T03:20:01.072 INFO:teuthology.orchestra.run.smithi033.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-10-20T03:20:01.075 INFO:teuthology.orchestra.run.smithi040.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-20T03:20:01.076 INFO:teuthology.orchestra.run.smithi033.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-20T03:20:01.077 INFO:teuthology.orchestra.run.smithi170.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-20T03:20:01.077 INFO:teuthology.orchestra.run.smithi040.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-20T03:20:01.077 INFO:teuthology.orchestra.run.smithi033.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-10-20T03:20:01.078 INFO:teuthology.orchestra.run.smithi033.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-20T03:20:01.078 INFO:teuthology.orchestra.run.smithi033.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-20T03:20:01.091 INFO:teuthology.orchestra.run.smithi170.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,016 kB] 2023-10-20T03:20:01.098 INFO:teuthology.orchestra.run.smithi170.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-20T03:20:01.099 INFO:teuthology.orchestra.run.smithi170.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [164 kB] 2023-10-20T03:20:01.100 INFO:teuthology.orchestra.run.smithi170.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-10-20T03:20:01.100 INFO:teuthology.orchestra.run.smithi170.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [793 kB] 2023-10-20T03:20:01.108 INFO:teuthology.orchestra.run.smithi170.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-10-20T03:20:01.110 INFO:teuthology.orchestra.run.smithi033.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-20T03:20:01.111 INFO:teuthology.orchestra.run.smithi033.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-20T03:20:01.116 INFO:teuthology.orchestra.run.smithi170.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-20T03:20:01.116 INFO:teuthology.orchestra.run.smithi170.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-10-20T03:20:01.117 INFO:teuthology.orchestra.run.smithi170.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-20T03:20:01.126 INFO:teuthology.orchestra.run.smithi170.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-20T03:20:01.127 INFO:teuthology.orchestra.run.smithi170.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-20T03:20:01.127 INFO:teuthology.orchestra.run.smithi170.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-20T03:20:01.140 INFO:teuthology.orchestra.run.smithi170.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [515 kB] 2023-10-20T03:20:01.146 INFO:teuthology.orchestra.run.smithi170.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [239 kB] 2023-10-20T03:20:01.149 INFO:teuthology.orchestra.run.smithi170.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-20T03:20:01.150 INFO:teuthology.orchestra.run.smithi170.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,036 kB] 2023-10-20T03:20:01.167 INFO:teuthology.orchestra.run.smithi170.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-10-20T03:20:01.167 INFO:teuthology.orchestra.run.smithi170.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [167 kB] 2023-10-20T03:20:01.170 INFO:teuthology.orchestra.run.smithi170.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-10-20T03:20:01.170 INFO:teuthology.orchestra.run.smithi170.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-10-20T03:20:01.176 INFO:teuthology.orchestra.run.smithi170.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [995 kB] 2023-10-20T03:20:01.186 INFO:teuthology.orchestra.run.smithi170.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-10-20T03:20:01.187 INFO:teuthology.orchestra.run.smithi170.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-20T03:20:01.188 INFO:teuthology.orchestra.run.smithi170.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-20T03:20:01.189 INFO:teuthology.orchestra.run.smithi170.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-20T03:20:01.189 INFO:teuthology.orchestra.run.smithi170.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-20T03:20:01.189 INFO:teuthology.orchestra.run.smithi170.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-20T03:20:01.207 INFO:teuthology.orchestra.run.smithi170.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-10-20T03:20:01.207 INFO:teuthology.orchestra.run.smithi170.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-10-20T03:20:01.208 INFO:teuthology.orchestra.run.smithi170.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-20T03:20:01.208 INFO:teuthology.orchestra.run.smithi170.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-20T03:20:01.227 INFO:teuthology.orchestra.run.smithi170.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-10-20T03:20:01.227 INFO:teuthology.orchestra.run.smithi170.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-10-20T03:20:01.227 INFO:teuthology.orchestra.run.smithi170.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-20T03:20:01.227 INFO:teuthology.orchestra.run.smithi170.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-10-20T03:20:01.234 INFO:teuthology.orchestra.run.smithi040.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-20T03:20:01.265 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-20T03:20:01.275 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [353 kB] 2023-10-20T03:20:01.280 INFO:teuthology.orchestra.run.smithi033.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-20T03:20:01.370 INFO:teuthology.orchestra.run.smithi040.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,103 kB] 2023-10-20T03:20:01.403 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [515 kB] 2023-10-20T03:20:01.416 INFO:teuthology.orchestra.run.smithi033.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [515 kB] 2023-10-20T03:20:01.542 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [896 kB] 2023-10-20T03:20:01.588 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,103 kB] 2023-10-20T03:20:01.600 INFO:teuthology.orchestra.run.smithi033.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,103 kB] 2023-10-20T03:20:01.642 INFO:teuthology.orchestra.run.smithi040.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [515 kB] 2023-10-20T03:20:01.681 INFO:teuthology.orchestra.run.smithi040.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [239 kB] 2023-10-20T03:20:01.688 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [180 kB] 2023-10-20T03:20:01.698 INFO:teuthology.orchestra.run.smithi040.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-20T03:20:01.700 INFO:teuthology.orchestra.run.smithi040.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-10-20T03:20:01.702 INFO:teuthology.orchestra.run.smithi040.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,036 kB] 2023-10-20T03:20:01.707 INFO:teuthology.orchestra.run.smithi064.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-20T03:20:01.708 INFO:teuthology.orchestra.run.smithi064.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-20T03:20:01.710 INFO:teuthology.orchestra.run.smithi064.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,016 kB] 2023-10-20T03:20:01.715 INFO:teuthology.orchestra.run.smithi064.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [239 kB] 2023-10-20T03:20:01.727 INFO:teuthology.orchestra.run.smithi033.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [239 kB] 2023-10-20T03:20:01.731 INFO:teuthology.orchestra.run.smithi064.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-20T03:20:01.732 INFO:teuthology.orchestra.run.smithi064.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-10-20T03:20:01.743 INFO:teuthology.orchestra.run.smithi033.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-20T03:20:01.746 INFO:teuthology.orchestra.run.smithi033.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,036 kB] 2023-10-20T03:20:01.753 INFO:teuthology.orchestra.run.smithi064.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,036 kB] 2023-10-20T03:20:01.753 INFO:teuthology.orchestra.run.smithi040.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [167 kB] 2023-10-20T03:20:01.759 INFO:teuthology.orchestra.run.smithi040.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-10-20T03:20:01.776 INFO:teuthology.orchestra.run.smithi064.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [164 kB] 2023-10-20T03:20:01.787 INFO:teuthology.orchestra.run.smithi064.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-10-20T03:20:01.787 INFO:teuthology.orchestra.run.smithi064.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [793 kB] 2023-10-20T03:20:01.787 INFO:teuthology.orchestra.run.smithi064.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [167 kB] 2023-10-20T03:20:01.792 INFO:teuthology.orchestra.run.smithi064.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-10-20T03:20:01.797 INFO:teuthology.orchestra.run.smithi033.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-10-20T03:20:01.799 INFO:teuthology.orchestra.run.smithi033.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [167 kB] 2023-10-20T03:20:01.805 INFO:teuthology.orchestra.run.smithi033.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-10-20T03:20:01.818 INFO:teuthology.orchestra.run.smithi064.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-10-20T03:20:01.923 INFO:teuthology.orchestra.run.smithi040.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-10-20T03:20:01.956 INFO:teuthology.orchestra.run.smithi064.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [995 kB] 2023-10-20T03:20:01.969 INFO:teuthology.orchestra.run.smithi033.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-10-20T03:20:01.998 INFO:teuthology.orchestra.run.smithi064.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-20T03:20:02.380 INFO:teuthology.orchestra.run.smithi064.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-10-20T03:20:02.387 INFO:teuthology.orchestra.run.smithi064.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-20T03:20:02.387 INFO:teuthology.orchestra.run.smithi064.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-20T03:20:02.402 INFO:teuthology.orchestra.run.smithi064.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-20T03:20:02.406 INFO:teuthology.orchestra.run.smithi064.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-20T03:20:02.493 INFO:teuthology.orchestra.run.smithi040.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [995 kB] 2023-10-20T03:20:02.540 INFO:teuthology.orchestra.run.smithi033.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [995 kB] 2023-10-20T03:20:02.576 INFO:teuthology.orchestra.run.smithi064.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-10-20T03:20:02.603 INFO:teuthology.orchestra.run.smithi040.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-10-20T03:20:02.614 INFO:teuthology.orchestra.run.smithi064.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-10-20T03:20:02.622 INFO:teuthology.orchestra.run.smithi040.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-20T03:20:02.624 INFO:teuthology.orchestra.run.smithi040.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-20T03:20:02.625 INFO:teuthology.orchestra.run.smithi040.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-20T03:20:02.627 INFO:teuthology.orchestra.run.smithi064.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-20T03:20:02.628 INFO:teuthology.orchestra.run.smithi040.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-20T03:20:02.628 INFO:teuthology.orchestra.run.smithi064.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-20T03:20:02.629 INFO:teuthology.orchestra.run.smithi064.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-20T03:20:02.629 INFO:teuthology.orchestra.run.smithi040.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-20T03:20:02.629 INFO:teuthology.orchestra.run.smithi040.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-10-20T03:20:02.630 INFO:teuthology.orchestra.run.smithi064.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-20T03:20:02.631 INFO:teuthology.orchestra.run.smithi064.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-20T03:20:02.632 INFO:teuthology.orchestra.run.smithi064.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-10-20T03:20:02.633 INFO:teuthology.orchestra.run.smithi040.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-10-20T03:20:02.638 INFO:teuthology.orchestra.run.smithi040.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-20T03:20:02.650 INFO:teuthology.orchestra.run.smithi033.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-10-20T03:20:02.658 INFO:teuthology.orchestra.run.smithi064.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-10-20T03:20:02.661 INFO:teuthology.orchestra.run.smithi064.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-20T03:20:02.669 INFO:teuthology.orchestra.run.smithi033.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-20T03:20:02.671 INFO:teuthology.orchestra.run.smithi033.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-20T03:20:02.674 INFO:teuthology.orchestra.run.smithi033.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-20T03:20:02.675 INFO:teuthology.orchestra.run.smithi033.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-20T03:20:02.676 INFO:teuthology.orchestra.run.smithi033.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-20T03:20:02.676 INFO:teuthology.orchestra.run.smithi033.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-10-20T03:20:02.680 INFO:teuthology.orchestra.run.smithi033.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-10-20T03:20:02.684 INFO:teuthology.orchestra.run.smithi033.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-20T03:20:02.803 INFO:teuthology.orchestra.run.smithi040.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-20T03:20:02.826 INFO:teuthology.orchestra.run.smithi064.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-20T03:20:02.849 INFO:teuthology.orchestra.run.smithi033.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-20T03:20:02.886 INFO:teuthology.orchestra.run.smithi040.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-10-20T03:20:02.908 INFO:teuthology.orchestra.run.smithi064.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-10-20T03:20:02.932 INFO:teuthology.orchestra.run.smithi033.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-10-20T03:20:02.987 INFO:teuthology.orchestra.run.smithi040.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-10-20T03:20:03.010 INFO:teuthology.orchestra.run.smithi064.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-10-20T03:20:03.013 INFO:teuthology.orchestra.run.smithi033.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-10-20T03:20:03.021 INFO:teuthology.orchestra.run.smithi040.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-20T03:20:03.044 INFO:teuthology.orchestra.run.smithi064.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-20T03:20:03.067 INFO:teuthology.orchestra.run.smithi033.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-20T03:20:03.076 INFO:teuthology.orchestra.run.smithi040.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-10-20T03:20:03.101 INFO:teuthology.orchestra.run.smithi064.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-10-20T03:20:03.124 INFO:teuthology.orchestra.run.smithi033.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-10-20T03:20:06.398 INFO:teuthology.orchestra.run.smithi170.stdout:Fetched 9,805 kB in 1s (6,762 kB/s) 2023-10-20T03:20:07.609 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 9,805 kB in 2s (3,990 kB/s) 2023-10-20T03:20:07.622 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 9,805 kB in 2s (3,975 kB/s) 2023-10-20T03:20:07.627 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 9,805 kB in 2s (3,976 kB/s) 2023-10-20T03:20:07.708 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2023-10-20T03:20:07.725 DEBUG:teuthology.orchestra.run.smithi170:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-20T03:20:07.783 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2023-10-20T03:20:07.969 INFO:teuthology.orchestra.run.smithi170.stdout:Building dependency tree... 2023-10-20T03:20:07.970 INFO:teuthology.orchestra.run.smithi170.stdout:Reading state information... 2023-10-20T03:20:08.125 INFO:teuthology.orchestra.run.smithi170.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T03:20:08.125 INFO:teuthology.orchestra.run.smithi170.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T03:20:08.126 INFO:teuthology.orchestra.run.smithi170.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T03:20:08.126 INFO:teuthology.orchestra.run.smithi170.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T03:20:08.127 INFO:teuthology.orchestra.run.smithi170.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-20T03:20:08.127 INFO:teuthology.orchestra.run.smithi170.stdout: python2.7-minimal 2023-10-20T03:20:08.127 INFO:teuthology.orchestra.run.smithi170.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T03:20:08.128 INFO:teuthology.orchestra.run.smithi170.stdout:The following additional packages will be installed: 2023-10-20T03:20:08.128 INFO:teuthology.orchestra.run.smithi170.stdout: linux-generic linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-20T03:20:08.128 INFO:teuthology.orchestra.run.smithi170.stdout: linux-headers-generic linux-image-5.15.0-87-generic 2023-10-20T03:20:08.128 INFO:teuthology.orchestra.run.smithi170.stdout: linux-modules-5.15.0-87-generic linux-modules-extra-5.15.0-87-generic 2023-10-20T03:20:08.129 INFO:teuthology.orchestra.run.smithi170.stdout:Suggested packages: 2023-10-20T03:20:08.129 INFO:teuthology.orchestra.run.smithi170.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-20T03:20:08.129 INFO:teuthology.orchestra.run.smithi170.stdout:Recommended packages: 2023-10-20T03:20:08.129 INFO:teuthology.orchestra.run.smithi170.stdout: thermald 2023-10-20T03:20:08.184 INFO:teuthology.orchestra.run.smithi170.stdout:The following NEW packages will be installed: 2023-10-20T03:20:08.184 INFO:teuthology.orchestra.run.smithi170.stdout: linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-20T03:20:08.184 INFO:teuthology.orchestra.run.smithi170.stdout: linux-image-5.15.0-87-generic linux-modules-5.15.0-87-generic 2023-10-20T03:20:08.185 INFO:teuthology.orchestra.run.smithi170.stdout: linux-modules-extra-5.15.0-87-generic 2023-10-20T03:20:08.185 INFO:teuthology.orchestra.run.smithi170.stdout:The following packages will be upgraded: 2023-10-20T03:20:08.186 INFO:teuthology.orchestra.run.smithi170.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-20T03:20:08.234 INFO:teuthology.orchestra.run.smithi170.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-10-20T03:20:08.235 INFO:teuthology.orchestra.run.smithi170.stdout:Need to get 113 MB of archives. 2023-10-20T03:20:08.235 INFO:teuthology.orchestra.run.smithi170.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-20T03:20:08.235 INFO:teuthology.orchestra.run.smithi170.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-87-generic amd64 5.15.0-87.97 [22.4 MB] 2023-10-20T03:20:08.633 INFO:teuthology.orchestra.run.smithi170.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-87-generic amd64 5.15.0-87.97 [11.5 MB] 2023-10-20T03:20:08.732 INFO:teuthology.orchestra.run.smithi170.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-87-generic amd64 5.15.0-87.97 [64.0 MB] 2023-10-20T03:20:08.897 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-20T03:20:08.905 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-10-20T03:20:08.909 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2023-10-20T03:20:08.915 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-20T03:20:08.923 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-20T03:20:08.927 DEBUG:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-20T03:20:08.981 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-20T03:20:08.989 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-10-20T03:20:08.991 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2023-10-20T03:20:09.170 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-20T03:20:09.170 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-20T03:20:09.179 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2023-10-20T03:20:09.179 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2023-10-20T03:20:09.180 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-10-20T03:20:09.181 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-10-20T03:20:09.337 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T03:20:09.337 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T03:20:09.338 INFO:teuthology.orchestra.run.smithi064.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T03:20:09.338 INFO:teuthology.orchestra.run.smithi064.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T03:20:09.339 INFO:teuthology.orchestra.run.smithi064.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-20T03:20:09.339 INFO:teuthology.orchestra.run.smithi064.stdout: python2.7-minimal 2023-10-20T03:20:09.339 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T03:20:09.340 INFO:teuthology.orchestra.run.smithi064.stdout:The following additional packages will be installed: 2023-10-20T03:20:09.340 INFO:teuthology.orchestra.run.smithi064.stdout: linux-generic linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-20T03:20:09.340 INFO:teuthology.orchestra.run.smithi064.stdout: linux-headers-generic linux-image-5.15.0-87-generic 2023-10-20T03:20:09.341 INFO:teuthology.orchestra.run.smithi064.stdout: linux-modules-5.15.0-87-generic linux-modules-extra-5.15.0-87-generic 2023-10-20T03:20:09.341 INFO:teuthology.orchestra.run.smithi064.stdout:Suggested packages: 2023-10-20T03:20:09.341 INFO:teuthology.orchestra.run.smithi064.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-20T03:20:09.341 INFO:teuthology.orchestra.run.smithi064.stdout:Recommended packages: 2023-10-20T03:20:09.342 INFO:teuthology.orchestra.run.smithi064.stdout: thermald 2023-10-20T03:20:09.344 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T03:20:09.345 INFO:teuthology.orchestra.run.smithi040.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T03:20:09.345 INFO:teuthology.orchestra.run.smithi040.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T03:20:09.345 INFO:teuthology.orchestra.run.smithi040.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T03:20:09.346 INFO:teuthology.orchestra.run.smithi040.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-20T03:20:09.346 INFO:teuthology.orchestra.run.smithi040.stdout: python2.7-minimal 2023-10-20T03:20:09.346 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T03:20:09.347 INFO:teuthology.orchestra.run.smithi040.stdout:The following additional packages will be installed: 2023-10-20T03:20:09.347 INFO:teuthology.orchestra.run.smithi040.stdout: linux-generic linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-20T03:20:09.347 INFO:teuthology.orchestra.run.smithi040.stdout: linux-headers-generic linux-image-5.15.0-87-generic 2023-10-20T03:20:09.348 INFO:teuthology.orchestra.run.smithi040.stdout: linux-modules-5.15.0-87-generic linux-modules-extra-5.15.0-87-generic 2023-10-20T03:20:09.348 INFO:teuthology.orchestra.run.smithi040.stdout:Suggested packages: 2023-10-20T03:20:09.348 INFO:teuthology.orchestra.run.smithi040.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-20T03:20:09.349 INFO:teuthology.orchestra.run.smithi040.stdout:Recommended packages: 2023-10-20T03:20:09.349 INFO:teuthology.orchestra.run.smithi040.stdout: thermald 2023-10-20T03:20:09.352 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T03:20:09.352 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T03:20:09.353 INFO:teuthology.orchestra.run.smithi033.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T03:20:09.353 INFO:teuthology.orchestra.run.smithi033.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T03:20:09.354 INFO:teuthology.orchestra.run.smithi033.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-20T03:20:09.354 INFO:teuthology.orchestra.run.smithi033.stdout: python2.7-minimal 2023-10-20T03:20:09.354 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T03:20:09.354 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2023-10-20T03:20:09.355 INFO:teuthology.orchestra.run.smithi033.stdout: linux-generic linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-20T03:20:09.355 INFO:teuthology.orchestra.run.smithi033.stdout: linux-headers-generic linux-image-5.15.0-87-generic 2023-10-20T03:20:09.355 INFO:teuthology.orchestra.run.smithi033.stdout: linux-modules-5.15.0-87-generic linux-modules-extra-5.15.0-87-generic 2023-10-20T03:20:09.356 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2023-10-20T03:20:09.356 INFO:teuthology.orchestra.run.smithi033.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-20T03:20:09.357 INFO:teuthology.orchestra.run.smithi033.stdout:Recommended packages: 2023-10-20T03:20:09.357 INFO:teuthology.orchestra.run.smithi033.stdout: thermald 2023-10-20T03:20:09.389 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2023-10-20T03:20:09.389 INFO:teuthology.orchestra.run.smithi064.stdout: linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-20T03:20:09.390 INFO:teuthology.orchestra.run.smithi064.stdout: linux-image-5.15.0-87-generic linux-modules-5.15.0-87-generic 2023-10-20T03:20:09.390 INFO:teuthology.orchestra.run.smithi064.stdout: linux-modules-extra-5.15.0-87-generic 2023-10-20T03:20:09.391 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages will be upgraded: 2023-10-20T03:20:09.391 INFO:teuthology.orchestra.run.smithi064.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-20T03:20:09.393 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2023-10-20T03:20:09.393 INFO:teuthology.orchestra.run.smithi033.stdout: linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-20T03:20:09.393 INFO:teuthology.orchestra.run.smithi033.stdout: linux-image-5.15.0-87-generic linux-modules-5.15.0-87-generic 2023-10-20T03:20:09.394 INFO:teuthology.orchestra.run.smithi033.stdout: linux-modules-extra-5.15.0-87-generic 2023-10-20T03:20:09.394 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages will be upgraded: 2023-10-20T03:20:09.395 INFO:teuthology.orchestra.run.smithi033.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-20T03:20:09.396 INFO:teuthology.orchestra.run.smithi040.stdout:The following NEW packages will be installed: 2023-10-20T03:20:09.396 INFO:teuthology.orchestra.run.smithi040.stdout: linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-20T03:20:09.396 INFO:teuthology.orchestra.run.smithi040.stdout: linux-image-5.15.0-87-generic linux-modules-5.15.0-87-generic 2023-10-20T03:20:09.396 INFO:teuthology.orchestra.run.smithi040.stdout: linux-modules-extra-5.15.0-87-generic 2023-10-20T03:20:09.397 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be upgraded: 2023-10-20T03:20:09.397 INFO:teuthology.orchestra.run.smithi040.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-20T03:20:09.441 INFO:teuthology.orchestra.run.smithi064.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-10-20T03:20:09.442 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 113 MB of archives. 2023-10-20T03:20:09.442 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-20T03:20:09.442 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-87-generic amd64 5.15.0-87.97 [22.4 MB] 2023-10-20T03:20:09.448 INFO:teuthology.orchestra.run.smithi040.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-10-20T03:20:09.448 INFO:teuthology.orchestra.run.smithi040.stdout:Need to get 113 MB of archives. 2023-10-20T03:20:09.448 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-20T03:20:09.448 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-87-generic amd64 5.15.0-87.97 [22.4 MB] 2023-10-20T03:20:09.450 INFO:teuthology.orchestra.run.smithi033.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-10-20T03:20:09.451 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 113 MB of archives. 2023-10-20T03:20:09.451 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-20T03:20:09.451 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-87-generic amd64 5.15.0-87.97 [22.4 MB] 2023-10-20T03:20:09.527 INFO:teuthology.orchestra.run.smithi170.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.87.84 [1,698 B] 2023-10-20T03:20:09.527 INFO:teuthology.orchestra.run.smithi170.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.87.84 [2,482 B] 2023-10-20T03:20:09.528 INFO:teuthology.orchestra.run.smithi170.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87 all 5.15.0-87.97 [12.3 MB] 2023-10-20T03:20:09.647 INFO:teuthology.orchestra.run.smithi170.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87-generic amd64 5.15.0-87.97 [2,888 kB] 2023-10-20T03:20:09.669 INFO:teuthology.orchestra.run.smithi170.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.87.84 [2,322 B] 2023-10-20T03:20:09.864 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-87-generic amd64 5.15.0-87.97 [11.5 MB] 2023-10-20T03:20:09.877 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-87-generic amd64 5.15.0-87.97 [11.5 MB] 2023-10-20T03:20:09.883 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-87-generic amd64 5.15.0-87.97 [11.5 MB] 2023-10-20T03:20:09.960 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-87-generic amd64 5.15.0-87.97 [64.0 MB] 2023-10-20T03:20:09.967 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-87-generic amd64 5.15.0-87.97 [64.0 MB] 2023-10-20T03:20:09.985 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-87-generic amd64 5.15.0-87.97 [64.0 MB] 2023-10-20T03:20:10.044 INFO:teuthology.orchestra.run.smithi170.stdout:Fetched 113 MB in 1s (76.5 MB/s) 2023-10-20T03:20:10.226 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package linux-modules-5.15.0-87-generic. 2023-10-20T03:20:10.759 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.87.84 [1,698 B] 2023-10-20T03:20:10.759 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.87.84 [2,482 B] 2023-10-20T03:20:10.760 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87 all 5.15.0-87.97 [12.3 MB] 2023-10-20T03:20:10.798 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.87.84 [1,698 B] 2023-10-20T03:20:10.798 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.87.84 [2,482 B] 2023-10-20T03:20:10.799 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87 all 5.15.0-87.97 [12.3 MB] 2023-10-20T03:20:10.883 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87-generic amd64 5.15.0-87.97 [2,888 kB] 2023-10-20T03:20:10.905 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.87.84 [2,322 B] 2023-10-20T03:20:10.911 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.87.84 [1,698 B] 2023-10-20T03:20:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87-generic amd64 5.15.0-87.97 [2,888 kB] 2023-10-20T03:20:10.933 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.87.84 [2,482 B] 2023-10-20T03:20:10.934 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87 all 5.15.0-87.97 [12.3 MB] 2023-10-20T03:20:10.951 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.87.84 [2,322 B] 2023-10-20T03:20:11.092 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87-generic amd64 5.15.0-87.97 [2,888 kB] 2023-10-20T03:20:11.125 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.87.84 [2,322 B] 2023-10-20T03:20:11.330 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 113 MB in 2s (75.1 MB/s) 2023-10-20T03:20:11.434 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 113 MB in 2s (69.5 MB/s) 2023-10-20T03:20:11.513 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-modules-5.15.0-87-generic. 2023-10-20T03:20:11.527 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 113 MB in 2s (65.6 MB/s) 2023-10-20T03:20:11.620 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-modules-5.15.0-87-generic. 2023-10-20T03:20:11.715 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-modules-5.15.0-87-generic. 2023-10-20T03:20:12.630 INFO:teuthology.orchestra.run.smithi170.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-20T03:20:12.634 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../0-linux-modules-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:12.932 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:13.567 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-20T03:20:13.571 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../0-linux-modules-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:13.650 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-20T03:20:13.655 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../0-linux-modules-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:13.718 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:13.725 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-20T03:20:13.729 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../0-linux-modules-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:13.796 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:13.826 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:15.058 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package linux-image-5.15.0-87-generic. 2023-10-20T03:20:15.079 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../1-linux-image-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:15.155 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:15.619 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-87-generic. 2023-10-20T03:20:15.640 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:15.677 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:15.811 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-image-5.15.0-87-generic. 2023-10-20T03:20:15.833 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../1-linux-image-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:15.835 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-image-5.15.0-87-generic. 2023-10-20T03:20:15.857 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../1-linux-image-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:15.898 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-image-5.15.0-87-generic. 2023-10-20T03:20:15.919 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../1-linux-image-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:15.928 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:15.935 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:15.996 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:16.363 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-87-generic. 2023-10-20T03:20:16.364 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-87-generic. 2023-10-20T03:20:16.384 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:16.385 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:16.421 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:16.422 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:16.443 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-87-generic. 2023-10-20T03:20:16.464 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:16.501 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:22.808 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../3-linux-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T03:20:22.846 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../3-linux-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T03:20:22.932 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T03:20:22.979 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking linux-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T03:20:23.056 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../3-linux-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T03:20:23.181 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T03:20:23.249 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T03:20:23.253 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../3-linux-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T03:20:23.304 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T03:20:23.369 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-image-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T03:20:23.375 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T03:20:23.424 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking linux-image-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T03:20:23.548 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T03:20:23.663 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-headers-5.15.0-87. 2023-10-20T03:20:23.668 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-image-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T03:20:23.686 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../5-linux-headers-5.15.0-87_5.15.0-87.97_all.deb ... 2023-10-20T03:20:23.717 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T03:20:23.730 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-20T03:20:23.744 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package linux-headers-5.15.0-87. 2023-10-20T03:20:23.762 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../5-linux-headers-5.15.0-87_5.15.0-87.97_all.deb ... 2023-10-20T03:20:23.811 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-20T03:20:23.904 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-image-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T03:20:23.965 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-headers-5.15.0-87. 2023-10-20T03:20:23.980 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../5-linux-headers-5.15.0-87_5.15.0-87.97_all.deb ... 2023-10-20T03:20:24.023 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-20T03:20:24.175 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-headers-5.15.0-87. 2023-10-20T03:20:24.197 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../5-linux-headers-5.15.0-87_5.15.0-87.97_all.deb ... 2023-10-20T03:20:24.241 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-20T03:20:29.186 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-headers-5.15.0-87-generic. 2023-10-20T03:20:29.223 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../6-linux-headers-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:29.261 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:29.318 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-headers-5.15.0-87-generic. 2023-10-20T03:20:29.356 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../6-linux-headers-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:29.393 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:29.564 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package linux-headers-5.15.0-87-generic. 2023-10-20T03:20:29.602 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../6-linux-headers-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:29.639 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:29.754 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-headers-5.15.0-87-generic. 2023-10-20T03:20:29.791 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../6-linux-headers-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-20T03:20:29.829 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:32.133 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T03:20:32.254 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-headers-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T03:20:32.256 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T03:20:32.355 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T03:20:32.411 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T03:20:32.534 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking linux-headers-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T03:20:32.582 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.87.84_amd64.deb ... 2023-10-20T03:20:32.596 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-20T03:20:32.655 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-20T03:20:32.709 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:32.715 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-headers-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-20T03:20:32.777 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:32.826 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-headers-generic (5.15.0.87.84) ... 2023-10-20T03:20:32.903 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-generic (5.15.0.87.84) ... 2023-10-20T03:20:32.926 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-20T03:20:32.929 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:33.005 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:33.057 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-20T03:20:33.064 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:33.165 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up linux-headers-generic (5.15.0.87.84) ... 2023-10-20T03:20:33.178 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:33.267 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:33.305 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-headers-generic (5.15.0.87.84) ... 2023-10-20T03:20:33.432 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:34.365 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:34.433 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:34.663 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:35.010 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:35.756 INFO:teuthology.orchestra.run.smithi033.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-20T03:20:35.757 INFO:teuthology.orchestra.run.smithi033.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-20T03:20:35.757 INFO:teuthology.orchestra.run.smithi033.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-87-generic 2023-10-20T03:20:35.757 INFO:teuthology.orchestra.run.smithi033.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:35.835 INFO:teuthology.orchestra.run.smithi064.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-20T03:20:35.836 INFO:teuthology.orchestra.run.smithi064.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-20T03:20:35.836 INFO:teuthology.orchestra.run.smithi064.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-87-generic 2023-10-20T03:20:35.836 INFO:teuthology.orchestra.run.smithi064.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:35.891 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:35.968 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:36.010 INFO:teuthology.orchestra.run.smithi170.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-20T03:20:36.010 INFO:teuthology.orchestra.run.smithi170.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-20T03:20:36.010 INFO:teuthology.orchestra.run.smithi170.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-87-generic 2023-10-20T03:20:36.010 INFO:teuthology.orchestra.run.smithi170.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:36.484 INFO:teuthology.orchestra.run.smithi040.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-20T03:20:36.484 INFO:teuthology.orchestra.run.smithi040.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-20T03:20:36.484 INFO:teuthology.orchestra.run.smithi040.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-87-generic 2023-10-20T03:20:36.485 INFO:teuthology.orchestra.run.smithi040.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:36.645 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:36.776 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:37.326 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-image-generic (5.15.0.87.84) ... 2023-10-20T03:20:37.394 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-image-generic (5.15.0.87.84) ... 2023-10-20T03:20:37.427 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-generic (5.15.0.87.84) ... 2023-10-20T03:20:37.496 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-generic (5.15.0.87.84) ... 2023-10-20T03:20:37.536 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:37.597 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:37.597 INFO:teuthology.orchestra.run.smithi033.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-20T03:20:37.597 INFO:teuthology.orchestra.run.smithi033.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:37.647 INFO:teuthology.orchestra.run.smithi064.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-20T03:20:37.648 INFO:teuthology.orchestra.run.smithi064.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:38.113 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-image-generic (5.15.0.87.84) ... 2023-10-20T03:20:38.202 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up linux-image-generic (5.15.0.87.84) ... 2023-10-20T03:20:38.239 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-generic (5.15.0.87.84) ... 2023-10-20T03:20:38.365 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:38.440 INFO:teuthology.orchestra.run.smithi040.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-20T03:20:38.440 INFO:teuthology.orchestra.run.smithi040.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:38.750 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up linux-generic (5.15.0.87.84) ... 2023-10-20T03:20:38.876 INFO:teuthology.orchestra.run.smithi170.stdout:Processing triggers for linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-20T03:20:38.954 INFO:teuthology.orchestra.run.smithi170.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-20T03:20:38.954 INFO:teuthology.orchestra.run.smithi170.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:52.459 INFO:teuthology.orchestra.run.smithi064.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-20T03:20:52.460 INFO:teuthology.orchestra.run.smithi064.stdout:Sourcing file `/etc/default/grub' 2023-10-20T03:20:52.477 INFO:teuthology.orchestra.run.smithi064.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-20T03:20:52.495 INFO:teuthology.orchestra.run.smithi064.stdout:Generating grub configuration file ... 2023-10-20T03:20:52.747 INFO:teuthology.orchestra.run.smithi064.stdout:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-20T03:20:52.765 INFO:teuthology.orchestra.run.smithi064.stdout:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:52.910 INFO:teuthology.orchestra.run.smithi033.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-20T03:20:52.911 INFO:teuthology.orchestra.run.smithi033.stdout:Sourcing file `/etc/default/grub' 2023-10-20T03:20:52.928 INFO:teuthology.orchestra.run.smithi033.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-20T03:20:52.956 INFO:teuthology.orchestra.run.smithi033.stdout:Generating grub configuration file ... 2023-10-20T03:20:52.978 INFO:teuthology.orchestra.run.smithi040.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-20T03:20:52.978 INFO:teuthology.orchestra.run.smithi040.stdout:Sourcing file `/etc/default/grub' 2023-10-20T03:20:52.985 INFO:teuthology.orchestra.run.smithi040.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-20T03:20:53.013 INFO:teuthology.orchestra.run.smithi040.stdout:Generating grub configuration file ... 2023-10-20T03:20:53.196 INFO:teuthology.orchestra.run.smithi033.stdout:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-20T03:20:53.220 INFO:teuthology.orchestra.run.smithi064.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-20T03:20:53.224 INFO:teuthology.orchestra.run.smithi033.stdout:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:53.226 INFO:teuthology.orchestra.run.smithi064.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-20T03:20:53.300 INFO:teuthology.orchestra.run.smithi064.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-20T03:20:53.305 INFO:teuthology.orchestra.run.smithi064.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-20T03:20:53.365 INFO:teuthology.orchestra.run.smithi040.stdout:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-20T03:20:53.374 INFO:teuthology.orchestra.run.smithi170.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-20T03:20:53.374 INFO:teuthology.orchestra.run.smithi170.stdout:Sourcing file `/etc/default/grub' 2023-10-20T03:20:53.384 INFO:teuthology.orchestra.run.smithi170.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-20T03:20:53.389 INFO:teuthology.orchestra.run.smithi040.stdout:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:53.414 INFO:teuthology.orchestra.run.smithi170.stdout:Generating grub configuration file ... 2023-10-20T03:20:53.474 INFO:teuthology.orchestra.run.smithi064.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-20T03:20:53.474 INFO:teuthology.orchestra.run.smithi064.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-20T03:20:53.474 INFO:teuthology.orchestra.run.smithi064.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-20T03:20:53.502 INFO:teuthology.orchestra.run.smithi033.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-20T03:20:53.508 INFO:teuthology.orchestra.run.smithi033.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-20T03:20:53.518 INFO:teuthology.orchestra.run.smithi064.stdout:done 2023-10-20T03:20:53.581 INFO:teuthology.orchestra.run.smithi033.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-20T03:20:53.586 INFO:teuthology.orchestra.run.smithi033.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-20T03:20:53.702 INFO:teuthology.orchestra.run.smithi170.stdout:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-20T03:20:53.720 INFO:teuthology.orchestra.run.smithi170.stdout:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:53.743 INFO:teuthology.orchestra.run.smithi033.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-20T03:20:53.743 INFO:teuthology.orchestra.run.smithi033.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-20T03:20:53.743 INFO:teuthology.orchestra.run.smithi033.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-20T03:20:53.755 INFO:teuthology.orchestra.run.smithi040.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-20T03:20:53.760 INFO:teuthology.orchestra.run.smithi040.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-20T03:20:53.787 INFO:teuthology.orchestra.run.smithi033.stdout:done 2023-10-20T03:20:53.829 INFO:teuthology.orchestra.run.smithi040.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-20T03:20:53.835 INFO:teuthology.orchestra.run.smithi040.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-20T03:20:53.996 INFO:teuthology.orchestra.run.smithi170.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-20T03:20:54.000 DEBUG:teuthology.orchestra.run.smithi064:> dpkg -s linux-image-generic 2023-10-20T03:20:54.001 INFO:teuthology.orchestra.run.smithi040.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-20T03:20:54.001 INFO:teuthology.orchestra.run.smithi040.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-20T03:20:54.001 INFO:teuthology.orchestra.run.smithi040.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-20T03:20:54.002 INFO:teuthology.orchestra.run.smithi170.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-20T03:20:54.020 INFO:teuthology.orchestra.run.smithi064.stdout:Package: linux-image-generic 2023-10-20T03:20:54.021 INFO:teuthology.orchestra.run.smithi064.stdout:Status: install ok installed 2023-10-20T03:20:54.021 INFO:teuthology.orchestra.run.smithi064.stdout:Priority: optional 2023-10-20T03:20:54.021 INFO:teuthology.orchestra.run.smithi064.stdout:Section: kernel 2023-10-20T03:20:54.021 INFO:teuthology.orchestra.run.smithi064.stdout:Installed-Size: 20 2023-10-20T03:20:54.021 INFO:teuthology.orchestra.run.smithi064.stdout:Maintainer: Ubuntu Kernel Team 2023-10-20T03:20:54.022 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: amd64 2023-10-20T03:20:54.022 INFO:teuthology.orchestra.run.smithi064.stdout:Source: linux-meta 2023-10-20T03:20:54.022 INFO:teuthology.orchestra.run.smithi064.stdout:Version: 5.15.0.87.84 2023-10-20T03:20:54.022 INFO:teuthology.orchestra.run.smithi064.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-20T03:20:54.022 INFO:teuthology.orchestra.run.smithi064.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-20T03:20:54.023 INFO:teuthology.orchestra.run.smithi064.stdout:Recommends: thermald 2023-10-20T03:20:54.023 INFO:teuthology.orchestra.run.smithi064.stdout:Description: Generic Linux kernel image 2023-10-20T03:20:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: This package will always depend on the latest generic kernel image 2023-10-20T03:20:54.023 INFO:teuthology.orchestra.run.smithi064.stdout: available. 2023-10-20T03:20:54.024 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-87-generic 2023-10-20T03:20:54.024 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-20T03:20:54.024 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-20T03:20:54.024 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2023-10-20T03:20:54.025 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2023-10-20T03:20:54.025 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get clean 2023-10-20T03:20:54.036 INFO:teuthology.orchestra.run.smithi040.stdout:done 2023-10-20T03:20:54.064 INFO:teuthology.orchestra.run.smithi170.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-20T03:20:54.069 INFO:teuthology.orchestra.run.smithi170.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-20T03:20:54.182 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2023-10-20T03:20:54.253 DEBUG:teuthology.orchestra.run.smithi033:> dpkg -s linux-image-generic 2023-10-20T03:20:54.254 INFO:teuthology.orchestra.run.smithi170.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-20T03:20:54.254 INFO:teuthology.orchestra.run.smithi170.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-20T03:20:54.255 INFO:teuthology.orchestra.run.smithi170.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-20T03:20:54.274 INFO:teuthology.orchestra.run.smithi033.stdout:Package: linux-image-generic 2023-10-20T03:20:54.274 INFO:teuthology.orchestra.run.smithi033.stdout:Status: install ok installed 2023-10-20T03:20:54.275 INFO:teuthology.orchestra.run.smithi033.stdout:Priority: optional 2023-10-20T03:20:54.275 INFO:teuthology.orchestra.run.smithi033.stdout:Section: kernel 2023-10-20T03:20:54.275 INFO:teuthology.orchestra.run.smithi033.stdout:Installed-Size: 20 2023-10-20T03:20:54.275 INFO:teuthology.orchestra.run.smithi033.stdout:Maintainer: Ubuntu Kernel Team 2023-10-20T03:20:54.276 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: amd64 2023-10-20T03:20:54.276 INFO:teuthology.orchestra.run.smithi033.stdout:Source: linux-meta 2023-10-20T03:20:54.276 INFO:teuthology.orchestra.run.smithi033.stdout:Version: 5.15.0.87.84 2023-10-20T03:20:54.276 INFO:teuthology.orchestra.run.smithi033.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-20T03:20:54.276 INFO:teuthology.orchestra.run.smithi033.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-20T03:20:54.277 INFO:teuthology.orchestra.run.smithi033.stdout:Recommends: thermald 2023-10-20T03:20:54.277 INFO:teuthology.orchestra.run.smithi033.stdout:Description: Generic Linux kernel image 2023-10-20T03:20:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: This package will always depend on the latest generic kernel image 2023-10-20T03:20:54.277 INFO:teuthology.orchestra.run.smithi033.stdout: available. 2023-10-20T03:20:54.277 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-87-generic 2023-10-20T03:20:54.278 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-20T03:20:54.278 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-20T03:20:54.278 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-20T03:20:54.278 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2023-10-20T03:20:54.279 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get clean 2023-10-20T03:20:54.301 INFO:teuthology.orchestra.run.smithi170.stdout:done 2023-10-20T03:20:54.442 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-20T03:20:54.443 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T03:20:54.456 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2023-10-20T03:20:54.505 DEBUG:teuthology.orchestra.run.smithi040:> dpkg -s linux-image-generic 2023-10-20T03:20:54.514 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-20T03:20:54.523 INFO:teuthology.orchestra.run.smithi040.stdout:Package: linux-image-generic 2023-10-20T03:20:54.523 INFO:teuthology.orchestra.run.smithi040.stdout:Status: install ok installed 2023-10-20T03:20:54.523 INFO:teuthology.orchestra.run.smithi040.stdout:Priority: optional 2023-10-20T03:20:54.524 INFO:teuthology.orchestra.run.smithi040.stdout:Section: kernel 2023-10-20T03:20:54.524 INFO:teuthology.orchestra.run.smithi040.stdout:Installed-Size: 20 2023-10-20T03:20:54.524 INFO:teuthology.orchestra.run.smithi040.stdout:Maintainer: Ubuntu Kernel Team 2023-10-20T03:20:54.524 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: amd64 2023-10-20T03:20:54.524 INFO:teuthology.orchestra.run.smithi040.stdout:Source: linux-meta 2023-10-20T03:20:54.525 INFO:teuthology.orchestra.run.smithi040.stdout:Version: 5.15.0.87.84 2023-10-20T03:20:54.525 INFO:teuthology.orchestra.run.smithi040.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-20T03:20:54.525 INFO:teuthology.orchestra.run.smithi040.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-20T03:20:54.525 INFO:teuthology.orchestra.run.smithi040.stdout:Recommends: thermald 2023-10-20T03:20:54.525 INFO:teuthology.orchestra.run.smithi040.stdout:Description: Generic Linux kernel image 2023-10-20T03:20:54.526 INFO:teuthology.orchestra.run.smithi040.stdout: This package will always depend on the latest generic kernel image 2023-10-20T03:20:54.526 INFO:teuthology.orchestra.run.smithi040.stdout: available. 2023-10-20T03:20:54.526 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-87-generic 2023-10-20T03:20:54.526 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-20T03:20:54.526 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-20T03:20:54.527 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-10-20T03:20:54.527 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=None, version=distro) 2023-10-20T03:20:54.527 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get clean 2023-10-20T03:20:54.589 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T03:20:54.596 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-20T03:20:54.598 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-20T03:20:54.615 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-20T03:20:54.708 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-20T03:20:54.713 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get update 2023-10-20T03:20:54.755 DEBUG:teuthology.orchestra.run.smithi170:> dpkg -s linux-image-generic 2023-10-20T03:20:54.776 INFO:teuthology.orchestra.run.smithi170.stdout:Package: linux-image-generic 2023-10-20T03:20:54.776 INFO:teuthology.orchestra.run.smithi170.stdout:Status: install ok installed 2023-10-20T03:20:54.776 INFO:teuthology.orchestra.run.smithi170.stdout:Priority: optional 2023-10-20T03:20:54.776 INFO:teuthology.orchestra.run.smithi170.stdout:Section: kernel 2023-10-20T03:20:54.776 INFO:teuthology.orchestra.run.smithi170.stdout:Installed-Size: 20 2023-10-20T03:20:54.777 INFO:teuthology.orchestra.run.smithi170.stdout:Maintainer: Ubuntu Kernel Team 2023-10-20T03:20:54.777 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: amd64 2023-10-20T03:20:54.777 INFO:teuthology.orchestra.run.smithi170.stdout:Source: linux-meta 2023-10-20T03:20:54.777 INFO:teuthology.orchestra.run.smithi170.stdout:Version: 5.15.0.87.84 2023-10-20T03:20:54.777 INFO:teuthology.orchestra.run.smithi170.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-20T03:20:54.778 INFO:teuthology.orchestra.run.smithi170.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-20T03:20:54.778 INFO:teuthology.orchestra.run.smithi170.stdout:Recommends: thermald 2023-10-20T03:20:54.778 INFO:teuthology.orchestra.run.smithi170.stdout:Description: Generic Linux kernel image 2023-10-20T03:20:54.778 INFO:teuthology.orchestra.run.smithi170.stdout: This package will always depend on the latest generic kernel image 2023-10-20T03:20:54.778 INFO:teuthology.orchestra.run.smithi170.stdout: available. 2023-10-20T03:20:54.779 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-87-generic 2023-10-20T03:20:54.779 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-20T03:20:54.780 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-20T03:20:54.780 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2023-10-20T03:20:54.780 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2023-10-20T03:20:54.780 DEBUG:teuthology.orchestra.run.smithi170:> sudo apt-get clean 2023-10-20T03:20:54.843 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T03:20:54.844 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-20T03:20:54.850 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-20T03:20:54.867 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-20T03:20:54.940 DEBUG:teuthology.orchestra.run.smithi170:> sudo apt-get update 2023-10-20T03:20:55.062 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T03:20:55.062 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-20T03:20:55.071 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-20T03:20:55.088 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-20T03:20:55.783 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-20T03:20:55.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-20T03:20:55.863 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-20T03:20:55.888 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-10-20T03:20:55.905 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-20T03:20:55.970 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-10-20T03:20:56.051 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-20T03:20:56.052 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-20T03:20:56.147 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2023-10-20T03:20:56.159 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-10-20T03:20:56.160 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-10-20T03:20:56.164 DEBUG:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-20T03:20:56.217 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2023-10-20T03:20:56.219 INFO:teuthology.orchestra.run.smithi064.stdout:linux-image-generic is already the newest version (5.15.0.87.84). 2023-10-20T03:20:56.219 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T03:20:56.219 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T03:20:56.221 INFO:teuthology.orchestra.run.smithi064.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T03:20:56.222 INFO:teuthology.orchestra.run.smithi064.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T03:20:56.222 INFO:teuthology.orchestra.run.smithi064.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-20T03:20:56.222 INFO:teuthology.orchestra.run.smithi064.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-20T03:20:56.222 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T03:20:56.260 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-10-20T03:20:56.262 DEBUG:teuthology.orchestra.run.smithi064:> dpkg -s linux-image-generic 2023-10-20T03:20:56.278 INFO:teuthology.orchestra.run.smithi064.stdout:Package: linux-image-generic 2023-10-20T03:20:56.278 INFO:teuthology.orchestra.run.smithi064.stdout:Status: install ok installed 2023-10-20T03:20:56.278 INFO:teuthology.orchestra.run.smithi064.stdout:Priority: optional 2023-10-20T03:20:56.278 INFO:teuthology.orchestra.run.smithi064.stdout:Section: kernel 2023-10-20T03:20:56.279 INFO:teuthology.orchestra.run.smithi064.stdout:Installed-Size: 20 2023-10-20T03:20:56.279 INFO:teuthology.orchestra.run.smithi064.stdout:Maintainer: Ubuntu Kernel Team 2023-10-20T03:20:56.279 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: amd64 2023-10-20T03:20:56.279 INFO:teuthology.orchestra.run.smithi064.stdout:Source: linux-meta 2023-10-20T03:20:56.279 INFO:teuthology.orchestra.run.smithi064.stdout:Version: 5.15.0.87.84 2023-10-20T03:20:56.280 INFO:teuthology.orchestra.run.smithi064.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-20T03:20:56.280 INFO:teuthology.orchestra.run.smithi064.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-20T03:20:56.280 INFO:teuthology.orchestra.run.smithi064.stdout:Recommends: thermald 2023-10-20T03:20:56.280 INFO:teuthology.orchestra.run.smithi064.stdout:Description: Generic Linux kernel image 2023-10-20T03:20:56.280 INFO:teuthology.orchestra.run.smithi064.stdout: This package will always depend on the latest generic kernel image 2023-10-20T03:20:56.280 INFO:teuthology.orchestra.run.smithi064.stdout: available. 2023-10-20T03:20:56.281 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2023-10-20T03:20:56.323 INFO:teuthology.orchestra.run.smithi033.stdout:linux-image-generic is already the newest version (5.15.0.87.84). 2023-10-20T03:20:56.324 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T03:20:56.324 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T03:20:56.324 INFO:teuthology.orchestra.run.smithi033.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T03:20:56.324 INFO:teuthology.orchestra.run.smithi033.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T03:20:56.325 INFO:teuthology.orchestra.run.smithi033.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-20T03:20:56.326 INFO:teuthology.orchestra.run.smithi033.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-20T03:20:56.326 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T03:20:56.326 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.MiaIYxJQi9 2023-10-20T03:20:56.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub/grub.cfg /tmp/tmp.MiaIYxJQi9 2023-10-20T03:20:56.346 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2023-10-20T03:20:56.364 DEBUG:teuthology.orchestra.run.smithi170:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-20T03:20:56.366 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-10-20T03:20:56.368 DEBUG:teuthology.orchestra.run.smithi033:> dpkg -s linux-image-generic 2023-10-20T03:20:56.382 INFO:teuthology.orchestra.run.smithi033.stdout:Package: linux-image-generic 2023-10-20T03:20:56.382 INFO:teuthology.orchestra.run.smithi033.stdout:Status: install ok installed 2023-10-20T03:20:56.382 INFO:teuthology.orchestra.run.smithi033.stdout:Priority: optional 2023-10-20T03:20:56.382 INFO:teuthology.orchestra.run.smithi033.stdout:Section: kernel 2023-10-20T03:20:56.382 INFO:teuthology.orchestra.run.smithi033.stdout:Installed-Size: 20 2023-10-20T03:20:56.383 INFO:teuthology.orchestra.run.smithi033.stdout:Maintainer: Ubuntu Kernel Team 2023-10-20T03:20:56.383 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: amd64 2023-10-20T03:20:56.383 INFO:teuthology.orchestra.run.smithi033.stdout:Source: linux-meta 2023-10-20T03:20:56.383 INFO:teuthology.orchestra.run.smithi033.stdout:Version: 5.15.0.87.84 2023-10-20T03:20:56.383 INFO:teuthology.orchestra.run.smithi033.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-20T03:20:56.384 INFO:teuthology.orchestra.run.smithi033.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-20T03:20:56.384 INFO:teuthology.orchestra.run.smithi033.stdout:Recommends: thermald 2023-10-20T03:20:56.384 INFO:teuthology.orchestra.run.smithi033.stdout:Description: Generic Linux kernel image 2023-10-20T03:20:56.384 INFO:teuthology.orchestra.run.smithi033.stdout: This package will always depend on the latest generic kernel image 2023-10-20T03:20:56.384 INFO:teuthology.orchestra.run.smithi033.stdout: available. 2023-10-20T03:20:56.385 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2023-10-20T03:20:56.386 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2023-10-20T03:20:56.386 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2023-10-20T03:20:56.387 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.MiaIYxJQi9 2023-10-20T03:20:56.425 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2023-10-20T03:20:56.426 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.s5OjnbmJRP 2023-10-20T03:20:56.427 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub/grub.cfg /tmp/tmp.s5OjnbmJRP 2023-10-20T03:20:56.482 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.s5OjnbmJRP 2023-10-20T03:20:56.507 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.MiaIYxJQi9 is 10KB 2023-10-20T03:20:56.520 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.MiaIYxJQi9 2023-10-20T03:20:56.526 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-20T03:20:56.535 INFO:teuthology.orchestra.run.smithi040.stdout:linux-image-generic is already the newest version (5.15.0.87.84). 2023-10-20T03:20:56.535 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T03:20:56.536 INFO:teuthology.orchestra.run.smithi040.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T03:20:56.536 INFO:teuthology.orchestra.run.smithi040.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T03:20:56.536 INFO:teuthology.orchestra.run.smithi040.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T03:20:56.536 INFO:teuthology.orchestra.run.smithi040.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-20T03:20:56.536 INFO:teuthology.orchestra.run.smithi040.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-20T03:20:56.537 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T03:20:56.573 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-10-20T03:20:56.575 DEBUG:teuthology.orchestra.run.smithi040:> dpkg -s linux-image-generic 2023-10-20T03:20:56.580 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:20:56.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-20T03:20:56.580 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-20T03:20:56.599 INFO:teuthology.orchestra.run.smithi170.stdout:Building dependency tree... 2023-10-20T03:20:56.600 INFO:teuthology.orchestra.run.smithi170.stdout:Reading state information... 2023-10-20T03:20:56.610 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.s5OjnbmJRP is 10KB 2023-10-20T03:20:56.622 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.s5OjnbmJRP 2023-10-20T03:20:56.627 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-20T03:20:56.629 INFO:teuthology.orchestra.run.smithi040.stdout:Package: linux-image-generic 2023-10-20T03:20:56.629 INFO:teuthology.orchestra.run.smithi040.stdout:Status: install ok installed 2023-10-20T03:20:56.629 INFO:teuthology.orchestra.run.smithi040.stdout:Priority: optional 2023-10-20T03:20:56.629 INFO:teuthology.orchestra.run.smithi040.stdout:Section: kernel 2023-10-20T03:20:56.629 INFO:teuthology.orchestra.run.smithi040.stdout:Installed-Size: 20 2023-10-20T03:20:56.630 INFO:teuthology.orchestra.run.smithi040.stdout:Maintainer: Ubuntu Kernel Team 2023-10-20T03:20:56.630 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: amd64 2023-10-20T03:20:56.630 INFO:teuthology.orchestra.run.smithi040.stdout:Source: linux-meta 2023-10-20T03:20:56.630 INFO:teuthology.orchestra.run.smithi040.stdout:Version: 5.15.0.87.84 2023-10-20T03:20:56.630 INFO:teuthology.orchestra.run.smithi040.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-20T03:20:56.631 INFO:teuthology.orchestra.run.smithi040.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-20T03:20:56.631 INFO:teuthology.orchestra.run.smithi040.stdout:Recommends: thermald 2023-10-20T03:20:56.631 INFO:teuthology.orchestra.run.smithi040.stdout:Description: Generic Linux kernel image 2023-10-20T03:20:56.631 INFO:teuthology.orchestra.run.smithi040.stdout: This package will always depend on the latest generic kernel image 2023-10-20T03:20:56.631 INFO:teuthology.orchestra.run.smithi040.stdout: available. 2023-10-20T03:20:56.632 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2023-10-20T03:20:56.646 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-87-generic 2023-10-20T03:20:56.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo update-grub 2023-10-20T03:20:56.673 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.5JO7z9wkS0 2023-10-20T03:20:56.674 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub/grub.cfg /tmp/tmp.5JO7z9wkS0 2023-10-20T03:20:56.683 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:20:56.683 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-20T03:20:56.683 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-20T03:20:56.738 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.5JO7z9wkS0 2023-10-20T03:20:56.746 INFO:teuthology.orchestra.run.smithi170.stdout:linux-image-generic is already the newest version (5.15.0.87.84). 2023-10-20T03:20:56.746 INFO:teuthology.orchestra.run.smithi170.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T03:20:56.746 INFO:teuthology.orchestra.run.smithi170.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T03:20:56.746 INFO:teuthology.orchestra.run.smithi170.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T03:20:56.747 INFO:teuthology.orchestra.run.smithi170.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T03:20:56.747 INFO:teuthology.orchestra.run.smithi170.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-20T03:20:56.747 INFO:teuthology.orchestra.run.smithi170.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-20T03:20:56.748 INFO:teuthology.orchestra.run.smithi170.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T03:20:56.751 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-87-generic 2023-10-20T03:20:56.751 DEBUG:teuthology.orchestra.run.smithi033:> sudo update-grub 2023-10-20T03:20:56.783 INFO:teuthology.orchestra.run.smithi170.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-10-20T03:20:56.785 DEBUG:teuthology.orchestra.run.smithi170:> dpkg -s linux-image-generic 2023-10-20T03:20:56.797 INFO:teuthology.orchestra.run.smithi170.stdout:Package: linux-image-generic 2023-10-20T03:20:56.798 INFO:teuthology.orchestra.run.smithi170.stdout:Status: install ok installed 2023-10-20T03:20:56.798 INFO:teuthology.orchestra.run.smithi170.stdout:Priority: optional 2023-10-20T03:20:56.798 INFO:teuthology.orchestra.run.smithi170.stdout:Section: kernel 2023-10-20T03:20:56.798 INFO:teuthology.orchestra.run.smithi170.stdout:Installed-Size: 20 2023-10-20T03:20:56.799 INFO:teuthology.orchestra.run.smithi170.stdout:Maintainer: Ubuntu Kernel Team 2023-10-20T03:20:56.799 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: amd64 2023-10-20T03:20:56.799 INFO:teuthology.orchestra.run.smithi170.stdout:Source: linux-meta 2023-10-20T03:20:56.799 INFO:teuthology.orchestra.run.smithi170.stdout:Version: 5.15.0.87.84 2023-10-20T03:20:56.799 INFO:teuthology.orchestra.run.smithi170.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-20T03:20:56.800 INFO:teuthology.orchestra.run.smithi170.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-20T03:20:56.800 INFO:teuthology.orchestra.run.smithi170.stdout:Recommends: thermald 2023-10-20T03:20:56.800 INFO:teuthology.orchestra.run.smithi170.stdout:Description: Generic Linux kernel image 2023-10-20T03:20:56.800 INFO:teuthology.orchestra.run.smithi170.stdout: This package will always depend on the latest generic kernel image 2023-10-20T03:20:56.801 INFO:teuthology.orchestra.run.smithi170.stdout: available. 2023-10-20T03:20:56.801 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-10-20T03:20:56.842 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.kFoxj9FCmd 2023-10-20T03:20:56.843 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub/grub.cfg /tmp/tmp.kFoxj9FCmd 2023-10-20T03:20:56.857 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.5JO7z9wkS0 is 10KB 2023-10-20T03:20:56.869 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.5JO7z9wkS0 2023-10-20T03:20:56.875 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-20T03:20:56.898 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.kFoxj9FCmd 2023-10-20T03:20:56.929 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:20:56.930 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-20T03:20:56.930 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-20T03:20:56.997 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-87-generic 2023-10-20T03:20:56.997 DEBUG:teuthology.orchestra.run.smithi040:> sudo update-grub 2023-10-20T03:20:57.004 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.kFoxj9FCmd is 10KB 2023-10-20T03:20:57.016 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.kFoxj9FCmd 2023-10-20T03:20:57.021 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-20T03:20:57.073 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:20:57.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-20T03:20:57.074 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-20T03:20:57.137 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-87-generic 2023-10-20T03:20:57.137 DEBUG:teuthology.orchestra.run.smithi170:> sudo update-grub 2023-10-20T03:20:57.770 INFO:teuthology.orchestra.run.smithi064.stderr:Sourcing file `/etc/default/grub' 2023-10-20T03:20:57.771 INFO:teuthology.orchestra.run.smithi064.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-20T03:20:57.776 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2023-10-20T03:20:57.830 INFO:teuthology.orchestra.run.smithi033.stderr:Sourcing file `/etc/default/grub' 2023-10-20T03:20:57.832 INFO:teuthology.orchestra.run.smithi033.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-20T03:20:57.834 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2023-10-20T03:20:57.925 INFO:teuthology.orchestra.run.smithi064.stderr:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-20T03:20:57.931 INFO:teuthology.orchestra.run.smithi064.stderr:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:58.009 INFO:teuthology.orchestra.run.smithi033.stderr:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-20T03:20:58.015 INFO:teuthology.orchestra.run.smithi033.stderr:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:58.134 INFO:teuthology.orchestra.run.smithi064.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-20T03:20:58.141 INFO:teuthology.orchestra.run.smithi064.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-20T03:20:58.181 INFO:teuthology.orchestra.run.smithi064.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-20T03:20:58.185 INFO:teuthology.orchestra.run.smithi064.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-20T03:20:58.186 INFO:teuthology.orchestra.run.smithi040.stderr:Sourcing file `/etc/default/grub' 2023-10-20T03:20:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-20T03:20:58.189 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2023-10-20T03:20:58.261 INFO:teuthology.orchestra.run.smithi033.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-20T03:20:58.266 INFO:teuthology.orchestra.run.smithi033.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-20T03:20:58.281 INFO:teuthology.orchestra.run.smithi064.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-20T03:20:58.281 INFO:teuthology.orchestra.run.smithi064.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-20T03:20:58.281 INFO:teuthology.orchestra.run.smithi064.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-20T03:20:58.300 INFO:teuthology.orchestra.run.smithi064.stderr:done 2023-10-20T03:20:58.313 INFO:teuthology.orchestra.run.smithi033.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-20T03:20:58.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2023-10-20T03:20:58.319 INFO:teuthology.orchestra.run.smithi033.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-20T03:20:58.436 INFO:teuthology.orchestra.run.smithi170.stderr:Sourcing file `/etc/default/grub' 2023-10-20T03:20:58.437 INFO:teuthology.orchestra.run.smithi170.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-20T03:20:58.441 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2023-10-20T03:20:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-20T03:20:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-20T03:20:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-20T03:20:58.460 INFO:teuthology.orchestra.run.smithi033.stderr:done 2023-10-20T03:20:58.469 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2023-10-20T03:20:58.472 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-20T03:20:58.478 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:58.616 INFO:teuthology.orchestra.run.smithi170.stderr:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-20T03:20:58.622 INFO:teuthology.orchestra.run.smithi170.stderr:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-20T03:20:58.764 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-20T03:20:58.769 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-20T03:20:58.816 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-20T03:20:58.822 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-20T03:20:58.938 INFO:teuthology.orchestra.run.smithi170.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-20T03:20:58.944 INFO:teuthology.orchestra.run.smithi170.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-20T03:20:58.964 INFO:teuthology.orchestra.run.smithi040.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-20T03:20:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-20T03:20:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-20T03:20:58.982 INFO:teuthology.orchestra.run.smithi170.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-20T03:20:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:done 2023-10-20T03:20:58.987 INFO:teuthology.orchestra.run.smithi170.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-20T03:20:58.994 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2023-10-20T03:20:59.134 INFO:teuthology.orchestra.run.smithi170.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-20T03:20:59.135 INFO:teuthology.orchestra.run.smithi170.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-20T03:20:59.135 INFO:teuthology.orchestra.run.smithi170.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-20T03:20:59.147 INFO:teuthology.orchestra.run.smithi170.stderr:done 2023-10-20T03:20:59.164 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2023-10-20T03:21:28.346 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2023-10-20T03:21:28.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-10-20T03:21:28.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:21:28.512 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-20T03:21:28.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-10-20T03:21:28.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:21:28.997 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-20T03:21:28.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-20T03:21:28.998 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:21:29.167 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2023-10-20T03:21:29.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-20T03:21:29.169 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:21:46.878 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2023-10-20T03:21:46.880 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-10-20T03:21:47.386 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-10-20T03:21:47.642 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2023-10-20T03:21:55.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-10-20T03:21:55.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:21:55.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-10-20T03:21:55.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:21:56.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-20T03:21:56.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:21:56.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-20T03:21:56.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:21:58.938 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2023-10-20T03:21:59.450 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-10-20T03:21:59.706 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2023-10-20T03:22:10.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-10-20T03:22:10.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:22:11.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-20T03:22:11.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:22:11.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-20T03:22:11.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:22:14.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-10-20T03:22:26.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-10-20T03:22:26.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:22:29.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-10-20T03:22:29.370 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2023-10-20T03:22:43.448 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-10-20T03:22:44.124 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-20T03:22:44.124 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.15.0-87-generic"... 2023-10-20T03:22:44.124 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-10-20T03:22:44.170 INFO:teuthology.orchestra.run.smithi170.stdout:5.15.0-87-generic 2023-10-20T03:22:44.171 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-87-generic vs 5.15.0-87-generic 2023-10-20T03:22:44.171 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-20T03:22:44.171 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-20T03:22:44.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-10-20T03:22:44.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:22:44.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-10-20T03:22:44.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:22:44.564 DEBUG:teuthology.orchestra.run.smithi033:> true 2023-10-20T03:22:45.154 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-10-20T03:22:45.154 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-87-generic"... 2023-10-20T03:22:45.154 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2023-10-20T03:22:45.172 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2023-10-20T03:22:45.172 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-20T03:22:45.200 INFO:teuthology.orchestra.run.smithi033.stdout:5.15.0-87-generic 2023-10-20T03:22:45.201 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-87-generic vs 5.15.0-87-generic 2023-10-20T03:22:45.201 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-20T03:22:45.202 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-20T03:22:45.241 DEBUG:teuthology.orchestra.run.smithi064:> true 2023-10-20T03:22:45.250 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2023-10-20T03:22:45.281 DEBUG:teuthology.parallel:result is None 2023-10-20T03:22:46.177 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-10-20T03:22:46.178 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.15.0-87-generic"... 2023-10-20T03:22:46.178 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2023-10-20T03:22:46.203 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-20T03:22:46.203 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-20T03:22:46.230 INFO:teuthology.orchestra.run.smithi064.stdout:5.15.0-87-generic 2023-10-20T03:22:46.230 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-87-generic vs 5.15.0-87-generic 2023-10-20T03:22:46.231 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-20T03:22:46.231 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-20T03:22:46.290 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2023-10-20T03:22:46.318 DEBUG:teuthology.parallel:result is None 2023-10-20T03:22:47.233 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2023-10-20T03:22:47.233 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-20T03:22:47.318 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2023-10-20T03:22:47.344 DEBUG:teuthology.parallel:result is None 2023-10-20T03:23:11.455 DEBUG:teuthology.orchestra.remote:timed out 2023-10-20T03:23:26.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-20T03:23:26.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:23:26.673 DEBUG:teuthology.orchestra.run.smithi040:> true 2023-10-20T03:23:27.218 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-20T03:23:27.218 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-87-generic"... 2023-10-20T03:23:27.219 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2023-10-20T03:23:27.264 INFO:teuthology.orchestra.run.smithi040.stdout:5.15.0-87-generic 2023-10-20T03:23:27.264 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-87-generic vs 5.15.0-87-generic 2023-10-20T03:23:27.264 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-20T03:23:27.265 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-20T03:23:28.266 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-20T03:23:28.267 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-20T03:23:28.330 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2023-10-20T03:23:28.370 DEBUG:teuthology.parallel:result is None 2023-10-20T03:23:28.370 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-20T03:23:28.383 INFO:teuthology.task.internal:Creating test directory... 2023-10-20T03:23:28.383 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-20T03:23:28.386 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-20T03:23:28.389 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-20T03:23:28.392 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-20T03:23:28.398 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-20T03:23:28.405 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-20T03:23:28.413 INFO:teuthology.task.internal:Creating archive directory... 2023-10-20T03:23:28.413 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-20T03:23:28.433 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-20T03:23:28.437 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-20T03:23:28.439 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-20T03:23:28.540 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-20T03:23:28.551 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-20T03:23:28.552 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-20T03:23:28.555 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-20T03:23:28.557 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-20T03:23:28.560 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-20T03:23:28.572 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-20T03:23:28.572 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-20T03:23:28.577 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-20T03:23:28.580 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-20T03:23:28.582 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-20T03:23:28.585 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-20T03:23:28.594 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-20T03:23:28.602 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-20T03:23:28.603 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-20T03:23:28.611 INFO:teuthology.task.internal:Configuring sudo... 2023-10-20T03:23:28.611 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-20T03:23:28.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-20T03:23:28.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-20T03:23:28.634 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-20T03:23:28.658 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-20T03:23:28.668 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-20T03:23:28.669 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-20T03:23:28.685 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-20T03:23:28.693 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-20T03:23:28.695 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-20T03:23:28.706 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-20T03:23:28.736 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-20T03:23:28.784 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:23:28.785 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-20T03:23:28.843 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-20T03:23:28.849 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-20T03:23:28.896 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:23:28.897 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-20T03:23:28.955 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-20T03:23:28.961 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-20T03:23:29.010 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:23:29.011 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-20T03:23:29.076 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-20T03:23:29.082 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-20T03:23:29.131 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:23:29.131 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-20T03:23:29.196 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2023-10-20T03:23:29.199 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2023-10-20T03:23:29.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2023-10-20T03:23:29.204 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2023-10-20T03:23:29.289 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-20T03:23:29.298 INFO:teuthology.task.internal:Starting timer... 2023-10-20T03:23:29.298 INFO:teuthology.run_tasks:Running task pcp... 2023-10-20T03:23:29.308 INFO:teuthology.run_tasks:Running task selinux... 2023-10-20T03:23:29.319 DEBUG:teuthology.task.selinux:Excluding smithi033: OS 'ubuntu' does not support SELinux 2023-10-20T03:23:29.320 DEBUG:teuthology.task.selinux:Excluding smithi040: OS 'ubuntu' does not support SELinux 2023-10-20T03:23:29.320 DEBUG:teuthology.task.selinux:Excluding smithi064: OS 'ubuntu' does not support SELinux 2023-10-20T03:23:29.320 DEBUG:teuthology.task.selinux:Excluding smithi170: OS 'ubuntu' does not support SELinux 2023-10-20T03:23:29.320 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-20T03:23:29.321 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-20T03:23:29.321 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-20T03:23:29.321 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-20T03:23:29.330 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-20T03:23:29.336 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-20T03:23:29.469 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-20T03:23:29.488 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-20T03:23:29.489 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi033.front.sepia.ceph.com,smithi040.front.sepia.ceph.com,smithi064.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-20T03:31:24.773 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi033.front.sepia.ceph.com'), Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2023-10-20T03:31:24.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-10-20T03:31:24.777 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:31:24.860 DEBUG:teuthology.orchestra.run.smithi033:> true 2023-10-20T03:31:24.978 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-10-20T03:31:24.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-20T03:31:24.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:31:25.058 DEBUG:teuthology.orchestra.run.smithi040:> true 2023-10-20T03:31:25.159 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-20T03:31:25.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-10-20T03:31:25.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:31:25.236 DEBUG:teuthology.orchestra.run.smithi064:> true 2023-10-20T03:31:25.358 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-10-20T03:31:25.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-20T03:31:25.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-20T03:31:25.435 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-10-20T03:31:25.530 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-10-20T03:31:25.532 INFO:teuthology.run_tasks:Running task clock... 2023-10-20T03:31:25.542 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-20T03:31:25.543 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-20T03:31:25.543 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-20T03:31:25.546 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-20T03:31:25.546 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-20T03:31:25.549 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-20T03:31:25.549 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-20T03:31:25.551 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-20T03:31:25.552 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-20T03:31:25.572 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-20T03:31:25.573 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: Command line: ntpd -gq 2023-10-20T03:31:25.573 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: ---------------------------------------------------- 2023-10-20T03:31:25.573 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: ntp-4 is maintained by Network Time Foundation, 2023-10-20T03:31:25.573 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-20T03:31:25.574 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: corporation. Support and training for ntp-4 are 2023-10-20T03:31:25.574 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: available at https://www.nwtime.org/support 2023-10-20T03:31:25.574 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: ---------------------------------------------------- 2023-10-20T03:31:25.574 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: proto: precision = 0.045 usec (-24) 2023-10-20T03:31:25.574 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: basedate set to 2022-02-04 2023-10-20T03:31:25.575 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: gps base set to 2022-02-06 (week 2196) 2023-10-20T03:31:25.575 INFO:teuthology.orchestra.run.smithi033.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-20T03:31:25.575 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-20T03:31:25.576 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: restrict ::: KOD does nothing without LIMITED. 2023-10-20T03:31:25.576 INFO:teuthology.orchestra.run.smithi033.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-20T03:31:25.577 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: Listen and drop on 0 v6wildcard [::]:123 2023-10-20T03:31:25.577 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-20T03:31:25.578 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: Listen normally on 2 lo 127.0.0.1:123 2023-10-20T03:31:25.578 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: Listen normally on 3 ens1f0 172.21.15.33:123 2023-10-20T03:31:25.578 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: Listen normally on 4 lo [::1]:123 2023-10-20T03:31:25.578 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:d294%4]:123 2023-10-20T03:31:25.579 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:25 ntpd[27927]: Listening on routing socket on fd #22 for interface updates 2023-10-20T03:31:25.580 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-20T03:31:25.580 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: Command line: ntpd -gq 2023-10-20T03:31:25.580 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: ---------------------------------------------------- 2023-10-20T03:31:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: ntp-4 is maintained by Network Time Foundation, 2023-10-20T03:31:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-20T03:31:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: corporation. Support and training for ntp-4 are 2023-10-20T03:31:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: available at https://www.nwtime.org/support 2023-10-20T03:31:25.581 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: ---------------------------------------------------- 2023-10-20T03:31:25.582 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: proto: precision = 0.046 usec (-24) 2023-10-20T03:31:25.583 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: basedate set to 2022-02-04 2023-10-20T03:31:25.583 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: gps base set to 2022-02-06 (week 2196) 2023-10-20T03:31:25.583 INFO:teuthology.orchestra.run.smithi040.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-20T03:31:25.584 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-20T03:31:25.584 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: restrict ::: KOD does nothing without LIMITED. 2023-10-20T03:31:25.584 INFO:teuthology.orchestra.run.smithi040.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-20T03:31:25.585 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: Listen and drop on 0 v6wildcard [::]:123 2023-10-20T03:31:25.585 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-20T03:31:25.586 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: Listen normally on 2 lo 127.0.0.1:123 2023-10-20T03:31:25.586 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: Listen normally on 3 ens1f0 172.21.15.40:123 2023-10-20T03:31:25.586 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: Listen normally on 4 lo [::1]:123 2023-10-20T03:31:25.586 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:12cc%4]:123 2023-10-20T03:31:25.586 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:25 ntpd[27885]: Listening on routing socket on fd #22 for interface updates 2023-10-20T03:31:25.587 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-20T03:31:25.588 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: Command line: ntpd -gq 2023-10-20T03:31:25.588 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: ---------------------------------------------------- 2023-10-20T03:31:25.588 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: ntp-4 is maintained by Network Time Foundation, 2023-10-20T03:31:25.588 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-20T03:31:25.588 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: corporation. Support and training for ntp-4 are 2023-10-20T03:31:25.589 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: available at https://www.nwtime.org/support 2023-10-20T03:31:25.589 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: ---------------------------------------------------- 2023-10-20T03:31:25.589 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: proto: precision = 0.064 usec (-24) 2023-10-20T03:31:25.589 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: basedate set to 2022-02-04 2023-10-20T03:31:25.589 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: gps base set to 2022-02-06 (week 2196) 2023-10-20T03:31:25.590 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-20T03:31:25.590 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: restrict ::: KOD does nothing without LIMITED. 2023-10-20T03:31:25.590 INFO:teuthology.orchestra.run.smithi064.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-20T03:31:25.590 INFO:teuthology.orchestra.run.smithi064.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-20T03:31:25.591 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: Listen and drop on 0 v6wildcard [::]:123 2023-10-20T03:31:25.592 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-20T03:31:25.592 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: Listen normally on 2 lo 127.0.0.1:123 2023-10-20T03:31:25.592 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: Listen normally on 3 enp3s0f1 172.21.15.64:123 2023-10-20T03:31:25.592 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: Listen normally on 4 lo [::1]:123 2023-10-20T03:31:25.593 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8df%5]:123 2023-10-20T03:31:25.593 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:25 ntpd[27971]: Listening on routing socket on fd #22 for interface updates 2023-10-20T03:31:25.600 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-20T03:31:25.601 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: Command line: ntpd -gq 2023-10-20T03:31:25.601 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: ---------------------------------------------------- 2023-10-20T03:31:25.601 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: ntp-4 is maintained by Network Time Foundation, 2023-10-20T03:31:25.601 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-20T03:31:25.602 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: corporation. Support and training for ntp-4 are 2023-10-20T03:31:25.602 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: available at https://www.nwtime.org/support 2023-10-20T03:31:25.602 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: ---------------------------------------------------- 2023-10-20T03:31:25.603 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: proto: precision = 0.044 usec (-24) 2023-10-20T03:31:25.603 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: basedate set to 2022-02-04 2023-10-20T03:31:25.603 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: gps base set to 2022-02-06 (week 2196) 2023-10-20T03:31:25.603 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-20T03:31:25.604 INFO:teuthology.orchestra.run.smithi170.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-20T03:31:25.604 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: restrict ::: KOD does nothing without LIMITED. 2023-10-20T03:31:25.605 INFO:teuthology.orchestra.run.smithi170.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-20T03:31:25.605 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: Listen and drop on 0 v6wildcard [::]:123 2023-10-20T03:31:25.606 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-20T03:31:25.606 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: Listen normally on 2 lo 127.0.0.1:123 2023-10-20T03:31:25.606 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: Listen normally on 3 enp3s0f1 172.21.15.170:123 2023-10-20T03:31:25.606 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: Listen normally on 4 lo [::1]:123 2023-10-20T03:31:25.607 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7255%5]:123 2023-10-20T03:31:25.607 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:25 ntpd[28017]: Listening on routing socket on fd #22 for interface updates 2023-10-20T03:31:26.572 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:26 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:26.572 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:26 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:26.577 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:26 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:26 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:26 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:26 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:26.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:26 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:26 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:26 ntpd[27885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:26.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:26 ntpd[27885]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:26 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:26 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:26 ntpd[27885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:26.580 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:26 ntpd[27885]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:26.581 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:26 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:26.581 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:26 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:26.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:26 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:26.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:26 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:26.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:26 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:26.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:26 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:26.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:26 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:26.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:26 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:26.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:26 ntpd[28017]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:26.603 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:26 ntpd[28017]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:27.572 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:27 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:27.572 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:27 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:27.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:27 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:27.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:27 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:27.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:27 ntpd[27927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:27.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:27 ntpd[27927]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:27.581 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:27 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:27.581 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:27 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:27.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:27 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:27.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:27 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:27.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:27 ntpd[27971]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:27.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:27 ntpd[27971]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:28.572 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:28 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:28.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:28 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:28.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:28 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:28.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:28 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:28.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:28 ntpd[27927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:28.574 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:28 ntpd[27927]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:28.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:28 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:28.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:28 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:28.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:28 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:28.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:28 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:28.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:28 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:28.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:28 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:28.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:28 ntpd[27885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:28.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:28 ntpd[27885]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:28.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:28 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:28.580 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:28 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:28.580 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:28 ntpd[27885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:28.580 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:28 ntpd[27885]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:28 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:28.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:28 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:28.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:28 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:28.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:28 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:28.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:28 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:28.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:28 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:28.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:28 ntpd[27971]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:28.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:28 ntpd[27971]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:28.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:28 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:28.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:28 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:28.584 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:28 ntpd[27971]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:28.584 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:28 ntpd[27971]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:28.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:28 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:28.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:28 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:28.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:28 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:28.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:28 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:28.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:28 ntpd[28017]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:28.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:28 ntpd[28017]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:29.572 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:29 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:29.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:29 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:29.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:29 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:29.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:29 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:29.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:29 ntpd[27927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:29.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:29 ntpd[27927]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:29.577 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:29 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:29.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:29 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:29.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:29 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:29.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:29 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:29.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:29 ntpd[27885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:29.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:29 ntpd[27885]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:29.581 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:29 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:29.581 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:29 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:29.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:29 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:29.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:29 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:29.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:29 ntpd[27971]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:29.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:29 ntpd[27971]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:29.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:29 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:29.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:29 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:29.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:29 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:29.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:29 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:29.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:29 ntpd[28017]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:29.603 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:29 ntpd[28017]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:30.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:30 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:30 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:30 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.574 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:30 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.574 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:30 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.574 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:30 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.574 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:30 ntpd[27927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:30.574 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:30 ntpd[27927]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:30.575 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:30 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.575 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:30 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.575 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:30 ntpd[27927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:30.575 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:30 ntpd[27927]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:30 ntpd[27885]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:30.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:30 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:30 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:30 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:30 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:30 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:30 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:30 ntpd[27971]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:30.584 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:30 ntpd[27971]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:30.584 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:30 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.584 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:30 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.584 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:30 ntpd[27971]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:30.584 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:30 ntpd[27971]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:30.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:30 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:30 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:30 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:30 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:30 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:30 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.603 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:30 ntpd[28017]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:30.603 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:30 ntpd[28017]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:30.603 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:30 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:30.603 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:30 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:30.603 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:30 ntpd[28017]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:30.604 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:30 ntpd[28017]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:31.572 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:31 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:31.572 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:31 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:31.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:31 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:31.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:31 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:31.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:31 ntpd[27927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:31.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:31 ntpd[27927]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:31 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:31 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:31 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:31 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:31 ntpd[27885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:31.582 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:31 ntpd[27885]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:31.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:31 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:31.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:31 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:31.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:31 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:31.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:31 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:31.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:31 ntpd[27971]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:31.584 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:31 ntpd[27971]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:31.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:31 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:31.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:31 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:31.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:31 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:31.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:31 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:31.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:31 ntpd[28017]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:31.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:31 ntpd[28017]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:32.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:32 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:32 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:32 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.574 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:32 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.574 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:32 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.574 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:32 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.575 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:32 ntpd[27927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:32.575 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:32 ntpd[27927]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:32.575 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:32 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.575 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:32 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.575 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:32 ntpd[27927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:32.576 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:32 ntpd[27927]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:32.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:32 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:32 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.578 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:32 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:32 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:32 ntpd[27885]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:32 ntpd[27885]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:32 ntpd[27885]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:32.579 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:32 ntpd[27885]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:32.580 INFO:teuthology.orchestra.run.smithi040.stdout:20 Oct 03:31:32 ntpd[27885]: ntpd: time slew -0.000652 s 2023-10-20T03:31:32.580 INFO:teuthology.orchestra.run.smithi040.stdout:ntpd: time slew -0.000652s 2023-10-20T03:31:32.581 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:32 ntpd[27885]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-20T03:31:32.581 INFO:teuthology.orchestra.run.smithi040.stderr:20 Oct 03:31:32 ntpd[27885]: can't open /var/log/ntpstats/loopstats.20231020: Permission denied 2023-10-20T03:31:32.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:32 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:32 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:32 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:32 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:32 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:32 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:32 ntpd[27971]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:32 ntpd[27971]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:32.584 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:32 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.585 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:32 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.585 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:32 ntpd[27971]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:32.585 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:32 ntpd[27971]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:32.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:32 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.601 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:32 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:32 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:32 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:32 ntpd[28017]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:32.602 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:32 ntpd[28017]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:32.603 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:32 ntpd[28017]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:32.603 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:32 ntpd[28017]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:32.603 INFO:teuthology.orchestra.run.smithi170.stdout:20 Oct 03:31:32 ntpd[28017]: ntpd: time slew -0.001253 s 2023-10-20T03:31:32.603 INFO:teuthology.orchestra.run.smithi170.stdout:ntpd: time slew -0.001253s 2023-10-20T03:31:32.604 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:32 ntpd[28017]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-20T03:31:32.604 INFO:teuthology.orchestra.run.smithi170.stderr:20 Oct 03:31:32 ntpd[28017]: can't open /var/log/ntpstats/loopstats.20231020: Permission denied 2023-10-20T03:31:32.645 INFO:teuthology.orchestra.run.smithi040.stdout: remote refid st t when poll reach delay offset jitter 2023-10-20T03:31:32.645 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================== 2023-10-20T03:31:32.645 INFO:teuthology.orchestra.run.smithi040.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:32.646 INFO:teuthology.orchestra.run.smithi040.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:32.646 INFO:teuthology.orchestra.run.smithi040.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:32.646 INFO:teuthology.orchestra.run.smithi040.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:32.682 INFO:teuthology.orchestra.run.smithi170.stdout: remote refid st t when poll reach delay offset jitter 2023-10-20T03:31:32.683 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================== 2023-10-20T03:31:32.683 INFO:teuthology.orchestra.run.smithi170.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:32.683 INFO:teuthology.orchestra.run.smithi170.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:32.683 INFO:teuthology.orchestra.run.smithi170.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:32.683 INFO:teuthology.orchestra.run.smithi170.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:33.572 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:33 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:33.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:33 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:33.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:33 ntpd[27927]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:33.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:33 ntpd[27927]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:33.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:33 ntpd[27927]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:33.573 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:33 ntpd[27927]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:33.574 INFO:teuthology.orchestra.run.smithi033.stdout:20 Oct 03:31:33 ntpd[27927]: ntpd: time slew -0.000444 s 2023-10-20T03:31:33.574 INFO:teuthology.orchestra.run.smithi033.stdout:ntpd: time slew -0.000444s 2023-10-20T03:31:33.575 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:33 ntpd[27927]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-20T03:31:33.575 INFO:teuthology.orchestra.run.smithi033.stderr:20 Oct 03:31:33 ntpd[27927]: can't open /var/log/ntpstats/loopstats.20231020: Permission denied 2023-10-20T03:31:33.581 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:33 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:33.581 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:33 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:33.581 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:33 ntpd[27971]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-20T03:31:33.581 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:33 ntpd[27971]: can't open /var/log/ntpstats/rawstats.20231020: Permission denied 2023-10-20T03:31:33.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:33 ntpd[27971]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-20T03:31:33.582 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:33 ntpd[27971]: can't open /var/log/ntpstats/peerstats.20231020: Permission denied 2023-10-20T03:31:33.582 INFO:teuthology.orchestra.run.smithi064.stdout:20 Oct 03:31:33 ntpd[27971]: ntpd: time slew -0.001161 s 2023-10-20T03:31:33.583 INFO:teuthology.orchestra.run.smithi064.stdout:ntpd: time slew -0.001161s 2023-10-20T03:31:33.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:33 ntpd[27971]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-20T03:31:33.583 INFO:teuthology.orchestra.run.smithi064.stderr:20 Oct 03:31:33 ntpd[27971]: can't open /var/log/ntpstats/loopstats.20231020: Permission denied 2023-10-20T03:31:33.654 INFO:teuthology.orchestra.run.smithi033.stdout: remote refid st t when poll reach delay offset jitter 2023-10-20T03:31:33.655 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================== 2023-10-20T03:31:33.655 INFO:teuthology.orchestra.run.smithi033.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:33.655 INFO:teuthology.orchestra.run.smithi033.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:33.655 INFO:teuthology.orchestra.run.smithi033.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:33.656 INFO:teuthology.orchestra.run.smithi033.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:33.666 INFO:teuthology.orchestra.run.smithi064.stdout: remote refid st t when poll reach delay offset jitter 2023-10-20T03:31:33.666 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================== 2023-10-20T03:31:33.667 INFO:teuthology.orchestra.run.smithi064.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:33.667 INFO:teuthology.orchestra.run.smithi064.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:33.667 INFO:teuthology.orchestra.run.smithi064.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:33.667 INFO:teuthology.orchestra.run.smithi064.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-20T03:31:33.668 INFO:teuthology.run_tasks:Running task install... 2023-10-20T03:31:33.679 DEBUG:teuthology.task.install:project ceph 2023-10-20T03:31:33.679 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8a5df55d22d3638066184c241442ff00fdc0ebdd'}} 2023-10-20T03:31:33.680 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8a5df55d22d3638066184c241442ff00fdc0ebdd'} 2023-10-20T03:31:33.680 INFO:teuthology.task.install:Using flavor: default 2023-10-20T03:31:33.686 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-20T03:31:33.686 INFO:teuthology.task.install:extra packages: [] 2023-10-20T03:31:33.687 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-key list | grep Ceph 2023-10-20T03:31:33.687 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-key list | grep Ceph 2023-10-20T03:31:33.688 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-key list | grep Ceph 2023-10-20T03:31:33.688 DEBUG:teuthology.orchestra.run.smithi170:> sudo apt-key list | grep Ceph 2023-10-20T03:31:33.748 INFO:teuthology.orchestra.run.smithi170.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-20T03:31:33.753 INFO:teuthology.orchestra.run.smithi033.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-20T03:31:33.754 INFO:teuthology.orchestra.run.smithi040.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-20T03:31:33.774 INFO:teuthology.orchestra.run.smithi064.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-20T03:31:33.776 INFO:teuthology.orchestra.run.smithi170.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-20T03:31:33.776 INFO:teuthology.orchestra.run.smithi170.stdout:uid [ unknown] Ceph.com (release key) 2023-10-20T03:31:33.777 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-20T03:31:33.778 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8a5df55d22d3638066184c241442ff00fdc0ebdd 2023-10-20T03:31:33.782 INFO:teuthology.orchestra.run.smithi033.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-20T03:31:33.783 INFO:teuthology.orchestra.run.smithi033.stdout:uid [ unknown] Ceph.com (release key) 2023-10-20T03:31:33.783 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-20T03:31:33.783 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8a5df55d22d3638066184c241442ff00fdc0ebdd 2023-10-20T03:31:33.787 INFO:teuthology.orchestra.run.smithi040.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-20T03:31:33.787 INFO:teuthology.orchestra.run.smithi040.stdout:uid [ unknown] Ceph.com (release key) 2023-10-20T03:31:33.787 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-20T03:31:33.788 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8a5df55d22d3638066184c241442ff00fdc0ebdd 2023-10-20T03:31:33.802 INFO:teuthology.orchestra.run.smithi064.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-20T03:31:33.802 INFO:teuthology.orchestra.run.smithi064.stdout:uid [ unknown] Ceph.com (release key) 2023-10-20T03:31:33.803 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-20T03:31:33.803 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8a5df55d22d3638066184c241442ff00fdc0ebdd 2023-10-20T03:31:33.931 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default/ 2023-10-20T03:31:33.931 INFO:teuthology.task.install.deb:Package version is 18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:31:33.944 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default/ 2023-10-20T03:31:33.945 INFO:teuthology.task.install.deb:Package version is 18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:31:34.001 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default/ 2023-10-20T03:31:34.001 INFO:teuthology.task.install.deb:Package version is 18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:31:34.016 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default/ 2023-10-20T03:31:34.016 INFO:teuthology.task.install.deb:Package version is 18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:31:34.059 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:31:34.059 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-20T03:31:34.062 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:31:34.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-20T03:31:34.078 DEBUG:teuthology.orchestra.run.smithi170:> sudo apt-get update 2023-10-20T03:31:34.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2023-10-20T03:31:34.126 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:31:34.127 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-20T03:31:34.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get update 2023-10-20T03:31:34.183 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:31:34.183 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-20T03:31:34.198 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2023-10-20T03:31:34.243 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T03:31:34.244 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T03:31:34.254 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-20T03:31:34.255 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-20T03:31:34.258 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-20T03:31:34.270 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-20T03:31:34.272 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-20T03:31:34.308 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T03:31:34.308 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-20T03:31:34.318 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-20T03:31:34.335 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-20T03:31:34.361 INFO:teuthology.orchestra.run.smithi170.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy InRelease 2023-10-20T03:31:34.363 INFO:teuthology.orchestra.run.smithi064.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy InRelease 2023-10-20T03:31:34.367 INFO:teuthology.orchestra.run.smithi040.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy InRelease 2023-10-20T03:31:34.368 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-20T03:31:34.368 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-20T03:31:34.378 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-20T03:31:34.388 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-20T03:31:34.389 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-20T03:31:34.395 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-20T03:31:34.397 INFO:teuthology.orchestra.run.smithi170.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-20T03:31:34.399 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-20T03:31:34.412 INFO:teuthology.orchestra.run.smithi040.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-20T03:31:34.433 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,423 B] 2023-10-20T03:31:34.434 INFO:teuthology.orchestra.run.smithi170.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-20T03:31:34.437 INFO:teuthology.orchestra.run.smithi064.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-20T03:31:34.461 INFO:teuthology.orchestra.run.smithi040.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-20T03:31:34.470 INFO:teuthology.orchestra.run.smithi170.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,423 B] 2023-10-20T03:31:34.473 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-20T03:31:34.488 INFO:teuthology.orchestra.run.smithi033.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy InRelease 2023-10-20T03:31:34.513 INFO:teuthology.orchestra.run.smithi170.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-20T03:31:34.516 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,423 B] 2023-10-20T03:31:34.525 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-20T03:31:34.563 INFO:teuthology.orchestra.run.smithi033.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-20T03:31:34.590 INFO:teuthology.orchestra.run.smithi170.stdout:Fetched 28.6 kB in 0s (84.6 kB/s) 2023-10-20T03:31:34.599 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-20T03:31:34.606 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 28.6 kB in 0s (81.5 kB/s) 2023-10-20T03:31:34.643 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,423 B] 2023-10-20T03:31:34.661 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 28.6 kB in 0s (83.9 kB/s) 2023-10-20T03:31:34.741 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 28.6 kB in 0s (79.7 kB/s) 2023-10-20T03:31:35.416 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2023-10-20T03:31:35.432 DEBUG:teuthology.orchestra.run.smithi170:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6703-g8a5df55d-1jammy cephadm=18.0.0-6703-g8a5df55d-1jammy ceph-mds=18.0.0-6703-g8a5df55d-1jammy ceph-mgr=18.0.0-6703-g8a5df55d-1jammy ceph-common=18.0.0-6703-g8a5df55d-1jammy ceph-fuse=18.0.0-6703-g8a5df55d-1jammy ceph-test=18.0.0-6703-g8a5df55d-1jammy ceph-volume=18.0.0-6703-g8a5df55d-1jammy radosgw=18.0.0-6703-g8a5df55d-1jammy python3-rados=18.0.0-6703-g8a5df55d-1jammy python3-rgw=18.0.0-6703-g8a5df55d-1jammy python3-cephfs=18.0.0-6703-g8a5df55d-1jammy python3-rbd=18.0.0-6703-g8a5df55d-1jammy libcephfs2=18.0.0-6703-g8a5df55d-1jammy libcephfs-dev=18.0.0-6703-g8a5df55d-1jammy librados2=18.0.0-6703-g8a5df55d-1jammy librbd1=18.0.0-6703-g8a5df55d-1jammy rbd-fuse=18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:31:35.495 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2023-10-20T03:31:35.502 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-20T03:31:35.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6703-g8a5df55d-1jammy cephadm=18.0.0-6703-g8a5df55d-1jammy ceph-mds=18.0.0-6703-g8a5df55d-1jammy ceph-mgr=18.0.0-6703-g8a5df55d-1jammy ceph-common=18.0.0-6703-g8a5df55d-1jammy ceph-fuse=18.0.0-6703-g8a5df55d-1jammy ceph-test=18.0.0-6703-g8a5df55d-1jammy ceph-volume=18.0.0-6703-g8a5df55d-1jammy radosgw=18.0.0-6703-g8a5df55d-1jammy python3-rados=18.0.0-6703-g8a5df55d-1jammy python3-rgw=18.0.0-6703-g8a5df55d-1jammy python3-cephfs=18.0.0-6703-g8a5df55d-1jammy python3-rbd=18.0.0-6703-g8a5df55d-1jammy libcephfs2=18.0.0-6703-g8a5df55d-1jammy libcephfs-dev=18.0.0-6703-g8a5df55d-1jammy librados2=18.0.0-6703-g8a5df55d-1jammy librbd1=18.0.0-6703-g8a5df55d-1jammy rbd-fuse=18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:31:35.552 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2023-10-20T03:31:35.568 DEBUG:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6703-g8a5df55d-1jammy cephadm=18.0.0-6703-g8a5df55d-1jammy ceph-mds=18.0.0-6703-g8a5df55d-1jammy ceph-mgr=18.0.0-6703-g8a5df55d-1jammy ceph-common=18.0.0-6703-g8a5df55d-1jammy ceph-fuse=18.0.0-6703-g8a5df55d-1jammy ceph-test=18.0.0-6703-g8a5df55d-1jammy ceph-volume=18.0.0-6703-g8a5df55d-1jammy radosgw=18.0.0-6703-g8a5df55d-1jammy python3-rados=18.0.0-6703-g8a5df55d-1jammy python3-rgw=18.0.0-6703-g8a5df55d-1jammy python3-cephfs=18.0.0-6703-g8a5df55d-1jammy python3-rbd=18.0.0-6703-g8a5df55d-1jammy libcephfs2=18.0.0-6703-g8a5df55d-1jammy libcephfs-dev=18.0.0-6703-g8a5df55d-1jammy librados2=18.0.0-6703-g8a5df55d-1jammy librbd1=18.0.0-6703-g8a5df55d-1jammy rbd-fuse=18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:31:35.570 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2023-10-20T03:31:35.633 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2023-10-20T03:31:35.641 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-10-20T03:31:35.658 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6703-g8a5df55d-1jammy cephadm=18.0.0-6703-g8a5df55d-1jammy ceph-mds=18.0.0-6703-g8a5df55d-1jammy ceph-mgr=18.0.0-6703-g8a5df55d-1jammy ceph-common=18.0.0-6703-g8a5df55d-1jammy ceph-fuse=18.0.0-6703-g8a5df55d-1jammy ceph-test=18.0.0-6703-g8a5df55d-1jammy ceph-volume=18.0.0-6703-g8a5df55d-1jammy radosgw=18.0.0-6703-g8a5df55d-1jammy python3-rados=18.0.0-6703-g8a5df55d-1jammy python3-rgw=18.0.0-6703-g8a5df55d-1jammy python3-cephfs=18.0.0-6703-g8a5df55d-1jammy python3-rbd=18.0.0-6703-g8a5df55d-1jammy libcephfs2=18.0.0-6703-g8a5df55d-1jammy libcephfs-dev=18.0.0-6703-g8a5df55d-1jammy librados2=18.0.0-6703-g8a5df55d-1jammy librbd1=18.0.0-6703-g8a5df55d-1jammy rbd-fuse=18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:31:35.685 INFO:teuthology.orchestra.run.smithi170.stdout:Building dependency tree... 2023-10-20T03:31:35.686 INFO:teuthology.orchestra.run.smithi170.stdout:Reading state information... 2023-10-20T03:31:35.723 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-10-20T03:31:35.739 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2023-10-20T03:31:35.740 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2023-10-20T03:31:35.822 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2023-10-20T03:31:35.823 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2023-10-20T03:31:35.863 INFO:teuthology.orchestra.run.smithi170.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T03:31:35.863 INFO:teuthology.orchestra.run.smithi170.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T03:31:35.864 INFO:teuthology.orchestra.run.smithi170.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T03:31:35.864 INFO:teuthology.orchestra.run.smithi170.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T03:31:35.864 INFO:teuthology.orchestra.run.smithi170.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-20T03:31:35.865 INFO:teuthology.orchestra.run.smithi170.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-20T03:31:35.865 INFO:teuthology.orchestra.run.smithi170.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T03:31:35.867 INFO:teuthology.orchestra.run.smithi170.stdout:The following additional packages will be installed: 2023-10-20T03:31:35.868 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-20T03:31:35.868 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-20T03:31:35.868 INFO:teuthology.orchestra.run.smithi170.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-20T03:31:35.868 INFO:teuthology.orchestra.run.smithi170.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-20T03:31:35.869 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-20T03:31:35.869 INFO:teuthology.orchestra.run.smithi170.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-20T03:31:35.869 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-20T03:31:35.869 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-20T03:31:35.870 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-20T03:31:35.870 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-20T03:31:35.870 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-20T03:31:35.870 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-20T03:31:35.870 INFO:teuthology.orchestra.run.smithi170.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-20T03:31:35.871 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-20T03:31:35.871 INFO:teuthology.orchestra.run.smithi170.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-20T03:31:35.871 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-20T03:31:35.871 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-20T03:31:35.871 INFO:teuthology.orchestra.run.smithi170.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-20T03:31:35.872 INFO:teuthology.orchestra.run.smithi170.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-20T03:31:35.872 INFO:teuthology.orchestra.run.smithi170.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-20T03:31:35.872 INFO:teuthology.orchestra.run.smithi170.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-20T03:31:35.872 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-20T03:31:35.872 INFO:teuthology.orchestra.run.smithi170.stdout:Suggested packages: 2023-10-20T03:31:35.873 INFO:teuthology.orchestra.run.smithi170.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-20T03:31:35.873 INFO:teuthology.orchestra.run.smithi170.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-20T03:31:35.873 INFO:teuthology.orchestra.run.smithi170.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-20T03:31:35.873 INFO:teuthology.orchestra.run.smithi170.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-20T03:31:35.873 INFO:teuthology.orchestra.run.smithi170.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-20T03:31:35.874 INFO:teuthology.orchestra.run.smithi170.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-20T03:31:35.874 INFO:teuthology.orchestra.run.smithi170.stdout: python3-watchdog 2023-10-20T03:31:35.874 INFO:teuthology.orchestra.run.smithi170.stdout:Recommended packages: 2023-10-20T03:31:35.875 INFO:teuthology.orchestra.run.smithi170.stdout: btrfs-tools 2023-10-20T03:31:35.894 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T03:31:35.895 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T03:31:35.895 INFO:teuthology.orchestra.run.smithi064.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T03:31:35.896 INFO:teuthology.orchestra.run.smithi064.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T03:31:35.896 INFO:teuthology.orchestra.run.smithi064.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-20T03:31:35.896 INFO:teuthology.orchestra.run.smithi064.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-20T03:31:35.896 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T03:31:35.898 INFO:teuthology.orchestra.run.smithi064.stdout:The following additional packages will be installed: 2023-10-20T03:31:35.899 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-20T03:31:35.899 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-20T03:31:35.900 INFO:teuthology.orchestra.run.smithi064.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-20T03:31:35.900 INFO:teuthology.orchestra.run.smithi064.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-20T03:31:35.900 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-20T03:31:35.901 INFO:teuthology.orchestra.run.smithi064.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-20T03:31:35.901 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-20T03:31:35.901 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-20T03:31:35.902 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-20T03:31:35.902 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-20T03:31:35.902 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-20T03:31:35.903 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-20T03:31:35.903 INFO:teuthology.orchestra.run.smithi064.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-20T03:31:35.903 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-20T03:31:35.903 INFO:teuthology.orchestra.run.smithi064.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-20T03:31:35.904 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-20T03:31:35.904 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-20T03:31:35.904 INFO:teuthology.orchestra.run.smithi064.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-20T03:31:35.904 INFO:teuthology.orchestra.run.smithi064.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-20T03:31:35.904 INFO:teuthology.orchestra.run.smithi064.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-20T03:31:35.905 INFO:teuthology.orchestra.run.smithi064.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-20T03:31:35.905 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-20T03:31:35.905 INFO:teuthology.orchestra.run.smithi064.stdout:Suggested packages: 2023-10-20T03:31:35.905 INFO:teuthology.orchestra.run.smithi064.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-20T03:31:35.906 INFO:teuthology.orchestra.run.smithi064.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-20T03:31:35.906 INFO:teuthology.orchestra.run.smithi064.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-20T03:31:35.906 INFO:teuthology.orchestra.run.smithi064.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-20T03:31:35.906 INFO:teuthology.orchestra.run.smithi064.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-20T03:31:35.907 INFO:teuthology.orchestra.run.smithi064.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-20T03:31:35.907 INFO:teuthology.orchestra.run.smithi064.stdout: python3-watchdog 2023-10-20T03:31:35.907 INFO:teuthology.orchestra.run.smithi064.stdout:Recommended packages: 2023-10-20T03:31:35.907 INFO:teuthology.orchestra.run.smithi064.stdout: btrfs-tools 2023-10-20T03:31:35.912 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-10-20T03:31:35.913 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-10-20T03:31:35.966 INFO:teuthology.orchestra.run.smithi170.stdout:The following NEW packages will be installed: 2023-10-20T03:31:35.967 INFO:teuthology.orchestra.run.smithi170.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-20T03:31:35.967 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-20T03:31:35.967 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-20T03:31:35.967 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-20T03:31:35.968 INFO:teuthology.orchestra.run.smithi170.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-20T03:31:35.968 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-20T03:31:35.969 INFO:teuthology.orchestra.run.smithi170.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-20T03:31:35.969 INFO:teuthology.orchestra.run.smithi170.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-20T03:31:35.970 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-20T03:31:35.970 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-20T03:31:35.970 INFO:teuthology.orchestra.run.smithi170.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-20T03:31:35.970 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-20T03:31:35.970 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-20T03:31:35.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-20T03:31:35.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-20T03:31:35.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-20T03:31:35.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-20T03:31:35.971 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-20T03:31:35.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-20T03:31:35.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-20T03:31:35.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-20T03:31:35.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-20T03:31:35.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-20T03:31:35.972 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-20T03:31:35.973 INFO:teuthology.orchestra.run.smithi170.stdout: socat xmlstarlet 2023-10-20T03:31:35.993 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2023-10-20T03:31:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-20T03:31:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-20T03:31:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-20T03:31:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-20T03:31:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-20T03:31:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-20T03:31:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-20T03:31:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-20T03:31:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-20T03:31:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-20T03:31:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-20T03:31:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-20T03:31:35.997 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-20T03:31:35.997 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-20T03:31:35.997 INFO:teuthology.orchestra.run.smithi064.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-20T03:31:35.997 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-20T03:31:35.997 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-20T03:31:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-20T03:31:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-20T03:31:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-20T03:31:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-20T03:31:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-20T03:31:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-20T03:31:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-20T03:31:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: socat xmlstarlet 2023-10-20T03:31:36.001 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T03:31:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T03:31:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T03:31:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T03:31:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-20T03:31:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-20T03:31:36.002 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T03:31:36.002 INFO:teuthology.orchestra.run.smithi040.stdout:The following additional packages will be installed: 2023-10-20T03:31:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-20T03:31:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-20T03:31:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-20T03:31:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-20T03:31:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-20T03:31:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-20T03:31:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-20T03:31:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-20T03:31:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-20T03:31:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-20T03:31:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-20T03:31:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-20T03:31:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-20T03:31:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-20T03:31:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-20T03:31:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-20T03:31:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-20T03:31:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-20T03:31:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-20T03:31:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-20T03:31:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-20T03:31:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-20T03:31:36.007 INFO:teuthology.orchestra.run.smithi040.stdout:Suggested packages: 2023-10-20T03:31:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-20T03:31:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-20T03:31:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-20T03:31:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-20T03:31:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-20T03:31:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-20T03:31:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: python3-watchdog 2023-10-20T03:31:36.009 INFO:teuthology.orchestra.run.smithi040.stdout:Recommended packages: 2023-10-20T03:31:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: btrfs-tools 2023-10-20T03:31:36.029 INFO:teuthology.orchestra.run.smithi170.stdout:0 upgraded, 109 newly installed, 0 to remove and 257 not upgraded. 2023-10-20T03:31:36.030 INFO:teuthology.orchestra.run.smithi170.stdout:Need to get 184 MB of archives. 2023-10-20T03:31:36.030 INFO:teuthology.orchestra.run.smithi170.stdout:After this operation, 825 MB of additional disk space will be used. 2023-10-20T03:31:36.030 INFO:teuthology.orchestra.run.smithi170.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-20T03:31:36.059 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 109 newly installed, 0 to remove and 257 not upgraded. 2023-10-20T03:31:36.059 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 184 MB of archives. 2023-10-20T03:31:36.059 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 825 MB of additional disk space will be used. 2023-10-20T03:31:36.060 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-20T03:31:36.083 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-10-20T03:31:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-20T03:31:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-20T03:31:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-20T03:31:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-20T03:31:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-20T03:31:36.085 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-20T03:31:36.086 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2023-10-20T03:31:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-20T03:31:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-20T03:31:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-20T03:31:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-20T03:31:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-20T03:31:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-20T03:31:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-20T03:31:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-20T03:31:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-20T03:31:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-20T03:31:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-20T03:31:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-20T03:31:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-20T03:31:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-20T03:31:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-20T03:31:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-20T03:31:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-20T03:31:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-20T03:31:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-20T03:31:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-20T03:31:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-20T03:31:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-20T03:31:36.091 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2023-10-20T03:31:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-20T03:31:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-20T03:31:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-20T03:31:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-20T03:31:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-20T03:31:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-20T03:31:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: python3-watchdog 2023-10-20T03:31:36.092 INFO:teuthology.orchestra.run.smithi033.stdout:Recommended packages: 2023-10-20T03:31:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: btrfs-tools 2023-10-20T03:31:36.093 INFO:teuthology.orchestra.run.smithi170.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-20T03:31:36.103 INFO:teuthology.orchestra.run.smithi040.stdout:The following NEW packages will be installed: 2023-10-20T03:31:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-20T03:31:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-20T03:31:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-20T03:31:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-20T03:31:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-20T03:31:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-20T03:31:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-20T03:31:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-20T03:31:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-20T03:31:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-20T03:31:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-20T03:31:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-20T03:31:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-20T03:31:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-20T03:31:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-20T03:31:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-20T03:31:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-20T03:31:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-20T03:31:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-20T03:31:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-20T03:31:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-20T03:31:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-20T03:31:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-20T03:31:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-20T03:31:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: socat xmlstarlet 2023-10-20T03:31:36.114 INFO:teuthology.orchestra.run.smithi170.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6703-g8a5df55d-1jammy [3,451 kB] 2023-10-20T03:31:36.123 INFO:teuthology.orchestra.run.smithi170.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-20T03:31:36.125 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-20T03:31:36.126 INFO:teuthology.orchestra.run.smithi170.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-20T03:31:36.141 INFO:teuthology.orchestra.run.smithi170.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-20T03:31:36.165 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 109 newly installed, 0 to remove and 257 not upgraded. 2023-10-20T03:31:36.166 INFO:teuthology.orchestra.run.smithi040.stdout:Need to get 184 MB of archives. 2023-10-20T03:31:36.166 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 825 MB of additional disk space will be used. 2023-10-20T03:31:36.166 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-20T03:31:36.167 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-20T03:31:36.170 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-20T03:31:36.185 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2023-10-20T03:31:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-20T03:31:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-20T03:31:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-20T03:31:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-20T03:31:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-20T03:31:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-20T03:31:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-20T03:31:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-20T03:31:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-20T03:31:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-20T03:31:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-20T03:31:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-20T03:31:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-20T03:31:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-20T03:31:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-20T03:31:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-20T03:31:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-20T03:31:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-20T03:31:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-20T03:31:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-20T03:31:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-20T03:31:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-20T03:31:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-20T03:31:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-20T03:31:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: socat xmlstarlet 2023-10-20T03:31:36.191 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-20T03:31:36.195 INFO:teuthology.orchestra.run.smithi170.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-20T03:31:36.200 INFO:teuthology.orchestra.run.smithi170.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-20T03:31:36.212 INFO:teuthology.orchestra.run.smithi170.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-20T03:31:36.216 INFO:teuthology.orchestra.run.smithi170.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-20T03:31:36.216 INFO:teuthology.orchestra.run.smithi170.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-20T03:31:36.217 INFO:teuthology.orchestra.run.smithi170.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-20T03:31:36.218 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6703-g8a5df55d-1jammy [3,451 kB] 2023-10-20T03:31:36.222 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-20T03:31:36.225 INFO:teuthology.orchestra.run.smithi170.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-20T03:31:36.226 INFO:teuthology.orchestra.run.smithi170.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-20T03:31:36.227 INFO:teuthology.orchestra.run.smithi170.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-20T03:31:36.227 INFO:teuthology.orchestra.run.smithi170.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-20T03:31:36.233 INFO:teuthology.orchestra.run.smithi170.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-20T03:31:36.234 INFO:teuthology.orchestra.run.smithi170.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-20T03:31:36.245 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-20T03:31:36.250 INFO:teuthology.orchestra.run.smithi170.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-20T03:31:36.250 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-20T03:31:36.251 INFO:teuthology.orchestra.run.smithi170.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-20T03:31:36.251 INFO:teuthology.orchestra.run.smithi170.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-20T03:31:36.252 INFO:teuthology.orchestra.run.smithi170.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-20T03:31:36.254 INFO:teuthology.orchestra.run.smithi170.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-20T03:31:36.255 INFO:teuthology.orchestra.run.smithi170.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-20T03:31:36.255 INFO:teuthology.orchestra.run.smithi170.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-20T03:31:36.256 INFO:teuthology.orchestra.run.smithi170.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-20T03:31:36.258 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-20T03:31:36.261 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-20T03:31:36.262 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-20T03:31:36.266 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6703-g8a5df55d-1jammy [3,451 kB] 2023-10-20T03:31:36.267 INFO:teuthology.orchestra.run.smithi170.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-20T03:31:36.267 INFO:teuthology.orchestra.run.smithi170.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-20T03:31:36.267 INFO:teuthology.orchestra.run.smithi064.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-20T03:31:36.268 INFO:teuthology.orchestra.run.smithi064.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-20T03:31:36.268 INFO:teuthology.orchestra.run.smithi064.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-20T03:31:36.276 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-20T03:31:36.276 INFO:teuthology.orchestra.run.smithi064.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-20T03:31:36.277 INFO:teuthology.orchestra.run.smithi064.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-20T03:31:36.277 INFO:teuthology.orchestra.run.smithi064.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-20T03:31:36.278 INFO:teuthology.orchestra.run.smithi064.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-20T03:31:36.283 INFO:teuthology.orchestra.run.smithi170.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-20T03:31:36.285 INFO:teuthology.orchestra.run.smithi170.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-20T03:31:36.285 INFO:teuthology.orchestra.run.smithi170.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-20T03:31:36.286 INFO:teuthology.orchestra.run.smithi170.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-20T03:31:36.288 INFO:teuthology.orchestra.run.smithi064.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-20T03:31:36.289 INFO:teuthology.orchestra.run.smithi064.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-20T03:31:36.289 INFO:teuthology.orchestra.run.smithi064.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-20T03:31:36.291 INFO:teuthology.orchestra.run.smithi170.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-20T03:31:36.292 INFO:teuthology.orchestra.run.smithi170.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-20T03:31:36.292 INFO:teuthology.orchestra.run.smithi170.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-20T03:31:36.292 INFO:teuthology.orchestra.run.smithi170.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-20T03:31:36.300 INFO:teuthology.orchestra.run.smithi170.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-20T03:31:36.302 INFO:teuthology.orchestra.run.smithi170.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-20T03:31:36.308 INFO:teuthology.orchestra.run.smithi064.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-20T03:31:36.308 INFO:teuthology.orchestra.run.smithi064.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-20T03:31:36.309 INFO:teuthology.orchestra.run.smithi064.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-20T03:31:36.311 INFO:teuthology.orchestra.run.smithi170.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-20T03:31:36.312 INFO:teuthology.orchestra.run.smithi064.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-20T03:31:36.312 INFO:teuthology.orchestra.run.smithi064.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-20T03:31:36.313 INFO:teuthology.orchestra.run.smithi170.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-20T03:31:36.313 INFO:teuthology.orchestra.run.smithi064.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-20T03:31:36.314 INFO:teuthology.orchestra.run.smithi170.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-20T03:31:36.314 INFO:teuthology.orchestra.run.smithi064.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-20T03:31:36.316 INFO:teuthology.orchestra.run.smithi170.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-20T03:31:36.316 INFO:teuthology.orchestra.run.smithi170.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-20T03:31:36.319 INFO:teuthology.orchestra.run.smithi170.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-20T03:31:36.327 INFO:teuthology.orchestra.run.smithi170.stdout:Get:45 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6703-g8a5df55d-1jammy [3,131 kB] 2023-10-20T03:31:36.329 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-20T03:31:36.329 INFO:teuthology.orchestra.run.smithi064.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-20T03:31:36.330 INFO:teuthology.orchestra.run.smithi064.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-20T03:31:36.330 INFO:teuthology.orchestra.run.smithi064.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-20T03:31:36.333 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-20T03:31:36.344 INFO:teuthology.orchestra.run.smithi040.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-20T03:31:36.349 INFO:teuthology.orchestra.run.smithi064.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-20T03:31:36.350 INFO:teuthology.orchestra.run.smithi064.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-20T03:31:36.350 INFO:teuthology.orchestra.run.smithi064.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-20T03:31:36.351 INFO:teuthology.orchestra.run.smithi040.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-20T03:31:36.351 INFO:teuthology.orchestra.run.smithi040.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-20T03:31:36.351 INFO:teuthology.orchestra.run.smithi040.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-20T03:31:36.352 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 109 newly installed, 0 to remove and 257 not upgraded. 2023-10-20T03:31:36.352 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 184 MB of archives. 2023-10-20T03:31:36.352 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 825 MB of additional disk space will be used. 2023-10-20T03:31:36.352 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6703-g8a5df55d-1jammy [3,451 kB] 2023-10-20T03:31:36.354 INFO:teuthology.orchestra.run.smithi064.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-20T03:31:36.355 INFO:teuthology.orchestra.run.smithi064.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-20T03:31:36.355 INFO:teuthology.orchestra.run.smithi064.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-20T03:31:36.356 INFO:teuthology.orchestra.run.smithi064.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-20T03:31:36.358 INFO:teuthology.orchestra.run.smithi040.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-20T03:31:36.359 INFO:teuthology.orchestra.run.smithi040.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-20T03:31:36.359 INFO:teuthology.orchestra.run.smithi040.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-20T03:31:36.360 INFO:teuthology.orchestra.run.smithi040.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-20T03:31:36.366 INFO:teuthology.orchestra.run.smithi040.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-20T03:31:36.367 INFO:teuthology.orchestra.run.smithi040.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-20T03:31:36.367 INFO:teuthology.orchestra.run.smithi064.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-20T03:31:36.369 INFO:teuthology.orchestra.run.smithi064.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-20T03:31:36.370 INFO:teuthology.orchestra.run.smithi064.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-20T03:31:36.371 INFO:teuthology.orchestra.run.smithi170.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-20T03:31:36.372 INFO:teuthology.orchestra.run.smithi170.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-20T03:31:36.373 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-20T03:31:36.373 INFO:teuthology.orchestra.run.smithi170.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-20T03:31:36.374 INFO:teuthology.orchestra.run.smithi170.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-20T03:31:36.374 INFO:teuthology.orchestra.run.smithi170.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-20T03:31:36.383 INFO:teuthology.orchestra.run.smithi040.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-20T03:31:36.384 INFO:teuthology.orchestra.run.smithi040.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-20T03:31:36.384 INFO:teuthology.orchestra.run.smithi040.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-20T03:31:36.385 INFO:teuthology.orchestra.run.smithi170.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-20T03:31:36.385 INFO:teuthology.orchestra.run.smithi040.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-20T03:31:36.387 INFO:teuthology.orchestra.run.smithi040.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-20T03:31:36.387 INFO:teuthology.orchestra.run.smithi064.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-20T03:31:36.388 INFO:teuthology.orchestra.run.smithi040.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-20T03:31:36.388 INFO:teuthology.orchestra.run.smithi040.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-20T03:31:36.389 INFO:teuthology.orchestra.run.smithi064.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-20T03:31:36.389 INFO:teuthology.orchestra.run.smithi040.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-20T03:31:36.390 INFO:teuthology.orchestra.run.smithi064.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-20T03:31:36.391 INFO:teuthology.orchestra.run.smithi064.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-20T03:31:36.394 INFO:teuthology.orchestra.run.smithi064.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-20T03:31:36.400 INFO:teuthology.orchestra.run.smithi040.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-20T03:31:36.401 INFO:teuthology.orchestra.run.smithi040.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-20T03:31:36.401 INFO:teuthology.orchestra.run.smithi170.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-20T03:31:36.403 INFO:teuthology.orchestra.run.smithi170.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-20T03:31:36.403 INFO:teuthology.orchestra.run.smithi170.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-20T03:31:36.417 INFO:teuthology.orchestra.run.smithi040.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-20T03:31:36.418 INFO:teuthology.orchestra.run.smithi040.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-20T03:31:36.419 INFO:teuthology.orchestra.run.smithi170.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-20T03:31:36.419 INFO:teuthology.orchestra.run.smithi170.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-20T03:31:36.420 INFO:teuthology.orchestra.run.smithi040.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-20T03:31:36.420 INFO:teuthology.orchestra.run.smithi040.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-20T03:31:36.421 INFO:teuthology.orchestra.run.smithi170.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-20T03:31:36.421 INFO:teuthology.orchestra.run.smithi170.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-20T03:31:36.421 INFO:teuthology.orchestra.run.smithi170.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-20T03:31:36.421 INFO:teuthology.orchestra.run.smithi170.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-20T03:31:36.423 INFO:teuthology.orchestra.run.smithi170.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-20T03:31:36.424 INFO:teuthology.orchestra.run.smithi170.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-20T03:31:36.425 INFO:teuthology.orchestra.run.smithi040.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-20T03:31:36.425 INFO:teuthology.orchestra.run.smithi040.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-20T03:31:36.426 INFO:teuthology.orchestra.run.smithi040.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-20T03:31:36.426 INFO:teuthology.orchestra.run.smithi040.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-20T03:31:36.427 INFO:teuthology.orchestra.run.smithi170.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-20T03:31:36.434 INFO:teuthology.orchestra.run.smithi040.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-20T03:31:36.435 INFO:teuthology.orchestra.run.smithi040.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-20T03:31:36.440 INFO:teuthology.orchestra.run.smithi170.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-20T03:31:36.442 INFO:teuthology.orchestra.run.smithi170.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-20T03:31:36.442 INFO:teuthology.orchestra.run.smithi170.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-20T03:31:36.444 INFO:teuthology.orchestra.run.smithi170.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6703-g8a5df55d-1jammy [688 kB] 2023-10-20T03:31:36.445 INFO:teuthology.orchestra.run.smithi040.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-20T03:31:36.445 INFO:teuthology.orchestra.run.smithi170.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-20T03:31:36.446 INFO:teuthology.orchestra.run.smithi040.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-20T03:31:36.447 INFO:teuthology.orchestra.run.smithi040.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-20T03:31:36.448 INFO:teuthology.orchestra.run.smithi170.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-20T03:31:36.449 INFO:teuthology.orchestra.run.smithi064.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-20T03:31:36.449 INFO:teuthology.orchestra.run.smithi040.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-20T03:31:36.450 INFO:teuthology.orchestra.run.smithi064.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-20T03:31:36.451 INFO:teuthology.orchestra.run.smithi040.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-20T03:31:36.452 INFO:teuthology.orchestra.run.smithi064.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-20T03:31:36.452 INFO:teuthology.orchestra.run.smithi064.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-20T03:31:36.452 INFO:teuthology.orchestra.run.smithi064.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-20T03:31:36.453 INFO:teuthology.orchestra.run.smithi040.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-20T03:31:36.453 INFO:teuthology.orchestra.run.smithi170.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-20T03:31:36.453 INFO:teuthology.orchestra.run.smithi064.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-20T03:31:36.454 INFO:teuthology.orchestra.run.smithi170.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-20T03:31:36.457 INFO:teuthology.orchestra.run.smithi170.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-20T03:31:36.458 INFO:teuthology.orchestra.run.smithi170.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-20T03:31:36.458 INFO:teuthology.orchestra.run.smithi170.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-20T03:31:36.467 INFO:teuthology.orchestra.run.smithi170.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6703-g8a5df55d-1jammy [356 kB] 2023-10-20T03:31:36.471 INFO:teuthology.orchestra.run.smithi064.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-20T03:31:36.473 INFO:teuthology.orchestra.run.smithi064.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-20T03:31:36.473 INFO:teuthology.orchestra.run.smithi064.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-20T03:31:36.474 INFO:teuthology.orchestra.run.smithi170.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-20T03:31:36.475 INFO:teuthology.orchestra.run.smithi170.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-20T03:31:36.482 INFO:teuthology.orchestra.run.smithi170.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-20T03:31:36.482 INFO:teuthology.orchestra.run.smithi170.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-20T03:31:36.482 INFO:teuthology.orchestra.run.smithi170.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-20T03:31:36.484 INFO:teuthology.orchestra.run.smithi170.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-20T03:31:36.485 INFO:teuthology.orchestra.run.smithi170.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-20T03:31:36.487 INFO:teuthology.orchestra.run.smithi170.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6703-g8a5df55d-1jammy [32.8 kB] 2023-10-20T03:31:36.488 INFO:teuthology.orchestra.run.smithi064.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-20T03:31:36.488 INFO:teuthology.orchestra.run.smithi170.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6703-g8a5df55d-1jammy [180 kB] 2023-10-20T03:31:36.491 INFO:teuthology.orchestra.run.smithi064.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-20T03:31:36.492 INFO:teuthology.orchestra.run.smithi064.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-20T03:31:36.493 INFO:teuthology.orchestra.run.smithi064.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-20T03:31:36.494 INFO:teuthology.orchestra.run.smithi064.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-20T03:31:36.495 INFO:teuthology.orchestra.run.smithi170.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6703-g8a5df55d-1jammy [62.4 kB] 2023-10-20T03:31:36.495 INFO:teuthology.orchestra.run.smithi064.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-20T03:31:36.497 INFO:teuthology.orchestra.run.smithi170.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6703-g8a5df55d-1jammy [329 kB] 2023-10-20T03:31:36.507 INFO:teuthology.orchestra.run.smithi040.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-20T03:31:36.508 INFO:teuthology.orchestra.run.smithi040.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-20T03:31:36.509 INFO:teuthology.orchestra.run.smithi170.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6703-g8a5df55d-1jammy [6,222 kB] 2023-10-20T03:31:36.510 INFO:teuthology.orchestra.run.smithi040.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-20T03:31:36.510 INFO:teuthology.orchestra.run.smithi040.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-20T03:31:36.510 INFO:teuthology.orchestra.run.smithi040.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-20T03:31:36.511 INFO:teuthology.orchestra.run.smithi040.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-20T03:31:36.511 INFO:teuthology.orchestra.run.smithi064.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-20T03:31:36.512 INFO:teuthology.orchestra.run.smithi064.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-20T03:31:36.515 INFO:teuthology.orchestra.run.smithi064.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-20T03:31:36.516 INFO:teuthology.orchestra.run.smithi064.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-20T03:31:36.518 INFO:teuthology.orchestra.run.smithi064.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-20T03:31:36.528 INFO:teuthology.orchestra.run.smithi040.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-20T03:31:36.530 INFO:teuthology.orchestra.run.smithi040.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-20T03:31:36.530 INFO:teuthology.orchestra.run.smithi040.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-20T03:31:36.531 INFO:teuthology.orchestra.run.smithi064.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-20T03:31:36.534 INFO:teuthology.orchestra.run.smithi064.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-20T03:31:36.536 INFO:teuthology.orchestra.run.smithi064.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-20T03:31:36.541 INFO:teuthology.orchestra.run.smithi064.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-20T03:31:36.543 INFO:teuthology.orchestra.run.smithi064.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-20T03:31:36.545 INFO:teuthology.orchestra.run.smithi040.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-20T03:31:36.545 INFO:teuthology.orchestra.run.smithi040.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-20T03:31:36.545 INFO:teuthology.orchestra.run.smithi040.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-20T03:31:36.546 INFO:teuthology.orchestra.run.smithi040.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-20T03:31:36.546 INFO:teuthology.orchestra.run.smithi040.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-20T03:31:36.547 INFO:teuthology.orchestra.run.smithi040.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-20T03:31:36.554 INFO:teuthology.orchestra.run.smithi064.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-20T03:31:36.555 INFO:teuthology.orchestra.run.smithi064.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-20T03:31:36.555 INFO:teuthology.orchestra.run.smithi064.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-20T03:31:36.555 INFO:teuthology.orchestra.run.smithi064.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-20T03:31:36.555 INFO:teuthology.orchestra.run.smithi064.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-20T03:31:36.559 INFO:teuthology.orchestra.run.smithi040.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-20T03:31:36.559 INFO:teuthology.orchestra.run.smithi040.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-20T03:31:36.563 INFO:teuthology.orchestra.run.smithi040.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-20T03:31:36.570 INFO:teuthology.orchestra.run.smithi064.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-20T03:31:36.571 INFO:teuthology.orchestra.run.smithi064.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-20T03:31:36.571 INFO:teuthology.orchestra.run.smithi064.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-20T03:31:36.572 INFO:teuthology.orchestra.run.smithi064.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-20T03:31:36.573 INFO:teuthology.orchestra.run.smithi064.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-20T03:31:36.575 INFO:teuthology.orchestra.run.smithi040.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-20T03:31:36.577 INFO:teuthology.orchestra.run.smithi040.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-20T03:31:36.577 INFO:teuthology.orchestra.run.smithi040.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-20T03:31:36.580 INFO:teuthology.orchestra.run.smithi040.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-20T03:31:36.583 INFO:teuthology.orchestra.run.smithi040.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-20T03:31:36.587 INFO:teuthology.orchestra.run.smithi040.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-20T03:31:36.589 INFO:teuthology.orchestra.run.smithi040.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-20T03:31:36.592 INFO:teuthology.orchestra.run.smithi040.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-20T03:31:36.593 INFO:teuthology.orchestra.run.smithi040.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-20T03:31:36.594 INFO:teuthology.orchestra.run.smithi040.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-20T03:31:36.609 INFO:teuthology.orchestra.run.smithi040.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-20T03:31:36.610 INFO:teuthology.orchestra.run.smithi040.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-20T03:31:36.616 INFO:teuthology.orchestra.run.smithi040.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-20T03:31:36.617 INFO:teuthology.orchestra.run.smithi040.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-20T03:31:36.617 INFO:teuthology.orchestra.run.smithi040.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-20T03:31:36.618 INFO:teuthology.orchestra.run.smithi040.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-20T03:31:36.619 INFO:teuthology.orchestra.run.smithi040.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-20T03:31:36.649 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-20T03:31:36.770 INFO:teuthology.orchestra.run.smithi170.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6703-g8a5df55d-1jammy [112 kB] 2023-10-20T03:31:36.773 INFO:teuthology.orchestra.run.smithi170.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6703-g8a5df55d-1jammy [439 kB] 2023-10-20T03:31:36.788 INFO:teuthology.orchestra.run.smithi170.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6703-g8a5df55d-1jammy [22.6 MB] 2023-10-20T03:31:36.821 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-20T03:31:36.835 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-20T03:31:36.908 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-20T03:31:37.101 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-20T03:31:37.112 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-20T03:31:37.141 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-20T03:31:37.152 INFO:teuthology.orchestra.run.smithi033.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-20T03:31:37.153 INFO:teuthology.orchestra.run.smithi033.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-20T03:31:37.154 INFO:teuthology.orchestra.run.smithi033.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-20T03:31:37.332 INFO:teuthology.orchestra.run.smithi033.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-20T03:31:37.544 INFO:teuthology.orchestra.run.smithi033.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-20T03:31:37.558 INFO:teuthology.orchestra.run.smithi033.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-20T03:31:37.609 INFO:teuthology.orchestra.run.smithi064.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6703-g8a5df55d-1jammy [3,131 kB] 2023-10-20T03:31:37.676 INFO:teuthology.orchestra.run.smithi033.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-20T03:31:37.684 INFO:teuthology.orchestra.run.smithi033.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-20T03:31:37.695 INFO:teuthology.orchestra.run.smithi033.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-20T03:31:37.711 INFO:teuthology.orchestra.run.smithi033.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-20T03:31:37.725 INFO:teuthology.orchestra.run.smithi033.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-20T03:31:37.729 INFO:teuthology.orchestra.run.smithi033.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-20T03:31:37.734 INFO:teuthology.orchestra.run.smithi033.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-20T03:31:37.836 INFO:teuthology.orchestra.run.smithi033.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-20T03:31:37.851 INFO:teuthology.orchestra.run.smithi040.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6703-g8a5df55d-1jammy [3,131 kB] 2023-10-20T03:31:37.881 INFO:teuthology.orchestra.run.smithi033.stdout:Get:24 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6703-g8a5df55d-1jammy [3,131 kB] 2023-10-20T03:31:37.885 INFO:teuthology.orchestra.run.smithi170.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6703-g8a5df55d-1jammy [6,856 kB] 2023-10-20T03:31:38.010 INFO:teuthology.orchestra.run.smithi033.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-20T03:31:38.119 INFO:teuthology.orchestra.run.smithi170.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6703-g8a5df55d-1jammy [238 kB] 2023-10-20T03:31:38.125 INFO:teuthology.orchestra.run.smithi170.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6703-g8a5df55d-1jammy [128 kB] 2023-10-20T03:31:38.129 INFO:teuthology.orchestra.run.smithi170.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6703-g8a5df55d-1jammy [1,497 kB] 2023-10-20T03:31:38.176 INFO:teuthology.orchestra.run.smithi033.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-20T03:31:38.198 INFO:teuthology.orchestra.run.smithi033.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-20T03:31:38.207 INFO:teuthology.orchestra.run.smithi170.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6703-g8a5df55d-1jammy [9,862 kB] 2023-10-20T03:31:38.303 INFO:teuthology.orchestra.run.smithi064.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6703-g8a5df55d-1jammy [688 kB] 2023-10-20T03:31:38.331 INFO:teuthology.orchestra.run.smithi033.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-20T03:31:38.348 INFO:teuthology.orchestra.run.smithi033.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-20T03:31:38.355 INFO:teuthology.orchestra.run.smithi033.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-20T03:31:38.428 INFO:teuthology.orchestra.run.smithi033.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-20T03:31:38.446 INFO:teuthology.orchestra.run.smithi064.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6703-g8a5df55d-1jammy [356 kB] 2023-10-20T03:31:38.455 INFO:teuthology.orchestra.run.smithi033.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-20T03:31:38.463 INFO:teuthology.orchestra.run.smithi033.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-20T03:31:38.513 INFO:teuthology.orchestra.run.smithi064.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6703-g8a5df55d-1jammy [32.8 kB] 2023-10-20T03:31:38.517 INFO:teuthology.orchestra.run.smithi064.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6703-g8a5df55d-1jammy [180 kB] 2023-10-20T03:31:38.535 INFO:teuthology.orchestra.run.smithi064.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6703-g8a5df55d-1jammy [62.4 kB] 2023-10-20T03:31:38.544 INFO:teuthology.orchestra.run.smithi064.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6703-g8a5df55d-1jammy [329 kB] 2023-10-20T03:31:38.585 INFO:teuthology.orchestra.run.smithi033.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-20T03:31:38.586 INFO:teuthology.orchestra.run.smithi033.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-20T03:31:38.601 INFO:teuthology.orchestra.run.smithi033.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6703-g8a5df55d-1jammy [688 kB] 2023-10-20T03:31:38.605 INFO:teuthology.orchestra.run.smithi064.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6703-g8a5df55d-1jammy [6,222 kB] 2023-10-20T03:31:38.684 INFO:teuthology.orchestra.run.smithi033.stdout:Get:37 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6703-g8a5df55d-1jammy [356 kB] 2023-10-20T03:31:38.751 INFO:teuthology.orchestra.run.smithi033.stdout:Get:38 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6703-g8a5df55d-1jammy [32.8 kB] 2023-10-20T03:31:38.752 INFO:teuthology.orchestra.run.smithi033.stdout:Get:39 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6703-g8a5df55d-1jammy [180 kB] 2023-10-20T03:31:38.754 INFO:teuthology.orchestra.run.smithi033.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-20T03:31:38.763 INFO:teuthology.orchestra.run.smithi033.stdout:Get:41 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6703-g8a5df55d-1jammy [62.4 kB] 2023-10-20T03:31:38.781 INFO:teuthology.orchestra.run.smithi170.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6703-g8a5df55d-1jammy [31.7 MB] 2023-10-20T03:31:38.815 INFO:teuthology.orchestra.run.smithi033.stdout:Get:42 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6703-g8a5df55d-1jammy [329 kB] 2023-10-20T03:31:38.828 INFO:teuthology.orchestra.run.smithi033.stdout:Get:43 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6703-g8a5df55d-1jammy [6,222 kB] 2023-10-20T03:31:38.945 INFO:teuthology.orchestra.run.smithi033.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-20T03:31:39.100 INFO:teuthology.orchestra.run.smithi033.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-20T03:31:39.206 INFO:teuthology.orchestra.run.smithi033.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-20T03:31:39.228 INFO:teuthology.orchestra.run.smithi033.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-20T03:31:39.254 INFO:teuthology.orchestra.run.smithi040.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6703-g8a5df55d-1jammy [688 kB] 2023-10-20T03:31:39.259 INFO:teuthology.orchestra.run.smithi033.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-20T03:31:39.272 INFO:teuthology.orchestra.run.smithi033.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-20T03:31:39.313 INFO:teuthology.orchestra.run.smithi033.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-20T03:31:39.322 INFO:teuthology.orchestra.run.smithi033.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-20T03:31:39.351 INFO:teuthology.orchestra.run.smithi033.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-20T03:31:39.498 INFO:teuthology.orchestra.run.smithi040.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6703-g8a5df55d-1jammy [356 kB] 2023-10-20T03:31:39.526 INFO:teuthology.orchestra.run.smithi064.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6703-g8a5df55d-1jammy [112 kB] 2023-10-20T03:31:39.542 INFO:teuthology.orchestra.run.smithi064.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6703-g8a5df55d-1jammy [439 kB] 2023-10-20T03:31:39.581 INFO:teuthology.orchestra.run.smithi033.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-20T03:31:39.606 INFO:teuthology.orchestra.run.smithi064.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6703-g8a5df55d-1jammy [22.6 MB] 2023-10-20T03:31:39.619 INFO:teuthology.orchestra.run.smithi040.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6703-g8a5df55d-1jammy [32.8 kB] 2023-10-20T03:31:39.622 INFO:teuthology.orchestra.run.smithi040.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6703-g8a5df55d-1jammy [180 kB] 2023-10-20T03:31:39.684 INFO:teuthology.orchestra.run.smithi040.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6703-g8a5df55d-1jammy [62.4 kB] 2023-10-20T03:31:39.730 INFO:teuthology.orchestra.run.smithi040.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6703-g8a5df55d-1jammy [329 kB] 2023-10-20T03:31:39.748 INFO:teuthology.orchestra.run.smithi033.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-20T03:31:39.778 INFO:teuthology.orchestra.run.smithi033.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6703-g8a5df55d-1jammy [112 kB] 2023-10-20T03:31:39.780 INFO:teuthology.orchestra.run.smithi033.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6703-g8a5df55d-1jammy [439 kB] 2023-10-20T03:31:39.833 INFO:teuthology.orchestra.run.smithi040.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6703-g8a5df55d-1jammy [6,222 kB] 2023-10-20T03:31:39.838 INFO:teuthology.orchestra.run.smithi033.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6703-g8a5df55d-1jammy [22.6 MB] 2023-10-20T03:31:40.095 INFO:teuthology.orchestra.run.smithi033.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-20T03:31:40.112 INFO:teuthology.orchestra.run.smithi033.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-20T03:31:40.120 INFO:teuthology.orchestra.run.smithi033.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-20T03:31:40.183 INFO:teuthology.orchestra.run.smithi033.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-20T03:31:40.479 INFO:teuthology.orchestra.run.smithi033.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-20T03:31:40.489 INFO:teuthology.orchestra.run.smithi033.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-20T03:31:40.490 INFO:teuthology.orchestra.run.smithi033.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-20T03:31:40.558 INFO:teuthology.orchestra.run.smithi033.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-20T03:31:40.559 INFO:teuthology.orchestra.run.smithi033.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-20T03:31:40.561 INFO:teuthology.orchestra.run.smithi033.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-20T03:31:40.726 INFO:teuthology.orchestra.run.smithi033.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-20T03:31:40.916 INFO:teuthology.orchestra.run.smithi033.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-20T03:31:40.978 INFO:teuthology.orchestra.run.smithi033.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-20T03:31:41.014 INFO:teuthology.orchestra.run.smithi033.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-20T03:31:41.072 INFO:teuthology.orchestra.run.smithi033.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-20T03:31:41.194 INFO:teuthology.orchestra.run.smithi170.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6703-g8a5df55d-1jammy [14.0 kB] 2023-10-20T03:31:41.195 INFO:teuthology.orchestra.run.smithi170.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6703-g8a5df55d-1jammy [839 kB] 2023-10-20T03:31:41.211 INFO:teuthology.orchestra.run.smithi170.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6703-g8a5df55d-1jammy [2,146 kB] 2023-10-20T03:31:41.242 INFO:teuthology.orchestra.run.smithi033.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-20T03:31:41.280 INFO:teuthology.orchestra.run.smithi033.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-20T03:31:41.304 INFO:teuthology.orchestra.run.smithi033.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-20T03:31:41.312 INFO:teuthology.orchestra.run.smithi033.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-20T03:31:41.336 INFO:teuthology.orchestra.run.smithi170.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6703-g8a5df55d-1jammy [283 kB] 2023-10-20T03:31:41.351 INFO:teuthology.orchestra.run.smithi170.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6703-g8a5df55d-1jammy [138 kB] 2023-10-20T03:31:41.353 INFO:teuthology.orchestra.run.smithi033.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-20T03:31:41.356 INFO:teuthology.orchestra.run.smithi170.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6703-g8a5df55d-1jammy [1,906 kB] 2023-10-20T03:31:41.377 INFO:teuthology.orchestra.run.smithi033.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-20T03:31:41.503 INFO:teuthology.orchestra.run.smithi170.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6703-g8a5df55d-1jammy [8,625 kB] 2023-10-20T03:31:41.575 INFO:teuthology.orchestra.run.smithi033.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-20T03:31:42.058 INFO:teuthology.orchestra.run.smithi033.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-20T03:31:42.058 INFO:teuthology.orchestra.run.smithi033.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-20T03:31:42.069 INFO:teuthology.orchestra.run.smithi033.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-20T03:31:42.072 INFO:teuthology.orchestra.run.smithi033.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-20T03:31:42.103 INFO:teuthology.orchestra.run.smithi033.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-20T03:31:42.116 INFO:teuthology.orchestra.run.smithi033.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-20T03:31:42.122 INFO:teuthology.orchestra.run.smithi040.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6703-g8a5df55d-1jammy [112 kB] 2023-10-20T03:31:42.182 INFO:teuthology.orchestra.run.smithi040.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6703-g8a5df55d-1jammy [439 kB] 2023-10-20T03:31:42.189 INFO:teuthology.orchestra.run.smithi170.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6703-g8a5df55d-1jammy [14.2 kB] 2023-10-20T03:31:42.190 INFO:teuthology.orchestra.run.smithi170.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6703-g8a5df55d-1jammy [48.2 MB] 2023-10-20T03:31:42.244 INFO:teuthology.orchestra.run.smithi033.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-20T03:31:42.245 INFO:teuthology.orchestra.run.smithi033.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-20T03:31:42.247 INFO:teuthology.orchestra.run.smithi033.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-20T03:31:42.265 INFO:teuthology.orchestra.run.smithi033.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-20T03:31:42.340 INFO:teuthology.orchestra.run.smithi040.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6703-g8a5df55d-1jammy [22.6 MB] 2023-10-20T03:31:42.433 INFO:teuthology.orchestra.run.smithi033.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-20T03:31:42.794 INFO:teuthology.orchestra.run.smithi064.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6703-g8a5df55d-1jammy [6,856 kB] 2023-10-20T03:31:42.935 INFO:teuthology.orchestra.run.smithi033.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6703-g8a5df55d-1jammy [6,856 kB] 2023-10-20T03:31:43.648 INFO:teuthology.orchestra.run.smithi064.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6703-g8a5df55d-1jammy [238 kB] 2023-10-20T03:31:43.674 INFO:teuthology.orchestra.run.smithi064.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6703-g8a5df55d-1jammy [128 kB] 2023-10-20T03:31:43.690 INFO:teuthology.orchestra.run.smithi064.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6703-g8a5df55d-1jammy [1,497 kB] 2023-10-20T03:31:43.760 INFO:teuthology.orchestra.run.smithi033.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6703-g8a5df55d-1jammy [238 kB] 2023-10-20T03:31:43.779 INFO:teuthology.orchestra.run.smithi033.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6703-g8a5df55d-1jammy [128 kB] 2023-10-20T03:31:43.808 INFO:teuthology.orchestra.run.smithi033.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6703-g8a5df55d-1jammy [1,497 kB] 2023-10-20T03:31:43.885 INFO:teuthology.orchestra.run.smithi064.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6703-g8a5df55d-1jammy [9,862 kB] 2023-10-20T03:31:43.995 INFO:teuthology.orchestra.run.smithi033.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6703-g8a5df55d-1jammy [9,862 kB] 2023-10-20T03:31:45.241 INFO:teuthology.orchestra.run.smithi064.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6703-g8a5df55d-1jammy [31.7 MB] 2023-10-20T03:31:45.362 INFO:teuthology.orchestra.run.smithi033.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6703-g8a5df55d-1jammy [31.7 MB] 2023-10-20T03:31:46.232 INFO:teuthology.orchestra.run.smithi170.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6703-g8a5df55d-1jammy [120 kB] 2023-10-20T03:31:46.234 INFO:teuthology.orchestra.run.smithi170.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6703-g8a5df55d-1jammy [38.3 kB] 2023-10-20T03:31:46.306 INFO:teuthology.orchestra.run.smithi170.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6703-g8a5df55d-1jammy [12.3 MB] 2023-10-20T03:31:47.119 INFO:teuthology.orchestra.run.smithi170.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6703-g8a5df55d-1jammy [91.4 kB] 2023-10-20T03:31:47.668 INFO:teuthology.orchestra.run.smithi170.stdout:Fetched 184 MB in 11s (16.5 MB/s) 2023-10-20T03:31:47.779 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-20T03:31:47.835 INFO:teuthology.orchestra.run.smithi170.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165984 files and directories currently installed.) 2023-10-20T03:31:47.840 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-20T03:31:47.912 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-20T03:31:48.208 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-20T03:31:48.233 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-20T03:31:48.273 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-20T03:31:48.405 INFO:teuthology.orchestra.run.smithi040.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6703-g8a5df55d-1jammy [6,856 kB] 2023-10-20T03:31:48.602 INFO:teuthology.orchestra.run.smithi033.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6703-g8a5df55d-1jammy [14.0 kB] 2023-10-20T03:31:48.603 INFO:teuthology.orchestra.run.smithi033.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6703-g8a5df55d-1jammy [839 kB] 2023-10-20T03:31:48.619 INFO:teuthology.orchestra.run.smithi033.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6703-g8a5df55d-1jammy [2,146 kB] 2023-10-20T03:31:48.635 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-20T03:31:48.662 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-20T03:31:48.702 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-20T03:31:48.741 INFO:teuthology.orchestra.run.smithi064.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6703-g8a5df55d-1jammy [14.0 kB] 2023-10-20T03:31:48.742 INFO:teuthology.orchestra.run.smithi064.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6703-g8a5df55d-1jammy [839 kB] 2023-10-20T03:31:48.758 INFO:teuthology.orchestra.run.smithi064.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6703-g8a5df55d-1jammy [2,146 kB] 2023-10-20T03:31:48.765 INFO:teuthology.orchestra.run.smithi033.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6703-g8a5df55d-1jammy [283 kB] 2023-10-20T03:31:48.773 INFO:teuthology.orchestra.run.smithi033.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6703-g8a5df55d-1jammy [138 kB] 2023-10-20T03:31:48.803 INFO:teuthology.orchestra.run.smithi033.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6703-g8a5df55d-1jammy [1,906 kB] 2023-10-20T03:31:48.929 INFO:teuthology.orchestra.run.smithi064.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6703-g8a5df55d-1jammy [283 kB] 2023-10-20T03:31:48.954 INFO:teuthology.orchestra.run.smithi033.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6703-g8a5df55d-1jammy [8,625 kB] 2023-10-20T03:31:48.957 INFO:teuthology.orchestra.run.smithi064.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6703-g8a5df55d-1jammy [138 kB] 2023-10-20T03:31:48.973 INFO:teuthology.orchestra.run.smithi064.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6703-g8a5df55d-1jammy [1,906 kB] 2023-10-20T03:31:49.021 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-20T03:31:49.048 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-20T03:31:49.088 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-20T03:31:49.147 INFO:teuthology.orchestra.run.smithi064.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6703-g8a5df55d-1jammy [8,625 kB] 2023-10-20T03:31:49.266 INFO:teuthology.orchestra.run.smithi040.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6703-g8a5df55d-1jammy [238 kB] 2023-10-20T03:31:49.276 INFO:teuthology.orchestra.run.smithi040.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6703-g8a5df55d-1jammy [128 kB] 2023-10-20T03:31:49.313 INFO:teuthology.orchestra.run.smithi040.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6703-g8a5df55d-1jammy [1,497 kB] 2023-10-20T03:31:49.450 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-20T03:31:49.476 INFO:teuthology.orchestra.run.smithi040.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6703-g8a5df55d-1jammy [9,862 kB] 2023-10-20T03:31:49.477 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T03:31:49.517 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:31:49.693 INFO:teuthology.orchestra.run.smithi033.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6703-g8a5df55d-1jammy [14.2 kB] 2023-10-20T03:31:49.693 INFO:teuthology.orchestra.run.smithi033.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6703-g8a5df55d-1jammy [48.2 MB] 2023-10-20T03:31:49.920 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-20T03:31:49.946 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T03:31:49.950 INFO:teuthology.orchestra.run.smithi064.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6703-g8a5df55d-1jammy [14.2 kB] 2023-10-20T03:31:49.951 INFO:teuthology.orchestra.run.smithi064.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6703-g8a5df55d-1jammy [48.2 MB] 2023-10-20T03:31:49.986 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:31:50.348 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-20T03:31:50.374 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T03:31:50.414 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:31:50.664 INFO:teuthology.orchestra.run.smithi040.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6703-g8a5df55d-1jammy [31.7 MB] 2023-10-20T03:31:50.725 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-20T03:31:50.741 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-20T03:31:50.775 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-20T03:31:51.087 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package librados2. 2023-10-20T03:31:51.113 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../008-librados2_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:31:51.153 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking librados2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:31:51.614 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package librbd1. 2023-10-20T03:31:51.640 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../009-librbd1_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:31:51.681 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking librbd1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:31:52.117 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libcephfs2. 2023-10-20T03:31:52.143 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:31:52.175 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libcephfs2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:31:52.478 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-rados. 2023-10-20T03:31:52.504 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:31:52.544 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-rados (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:31:52.889 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-20T03:31:52.915 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:31:53.131 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-ceph-argparse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:31:53.375 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-cephfs. 2023-10-20T03:31:53.390 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:31:53.425 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-cephfs (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:31:53.598 INFO:teuthology.orchestra.run.smithi040.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6703-g8a5df55d-1jammy [14.0 kB] 2023-10-20T03:31:53.606 INFO:teuthology.orchestra.run.smithi040.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6703-g8a5df55d-1jammy [839 kB] 2023-10-20T03:31:53.626 INFO:teuthology.orchestra.run.smithi040.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6703-g8a5df55d-1jammy [2,146 kB] 2023-10-20T03:31:53.694 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-20T03:31:53.720 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:31:53.761 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:31:53.782 INFO:teuthology.orchestra.run.smithi040.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6703-g8a5df55d-1jammy [283 kB] 2023-10-20T03:31:53.821 INFO:teuthology.orchestra.run.smithi040.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6703-g8a5df55d-1jammy [138 kB] 2023-10-20T03:31:53.825 INFO:teuthology.orchestra.run.smithi040.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6703-g8a5df55d-1jammy [1,906 kB] 2023-10-20T03:31:53.861 INFO:teuthology.orchestra.run.smithi040.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6703-g8a5df55d-1jammy [8,625 kB] 2023-10-20T03:31:54.080 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-20T03:31:54.107 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-20T03:31:54.147 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-20T03:31:54.432 INFO:teuthology.orchestra.run.smithi040.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6703-g8a5df55d-1jammy [14.2 kB] 2023-10-20T03:31:54.432 INFO:teuthology.orchestra.run.smithi040.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6703-g8a5df55d-1jammy [48.2 MB] 2023-10-20T03:31:54.475 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-prettytable. 2023-10-20T03:31:54.501 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-20T03:31:54.541 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-20T03:31:54.794 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-rbd. 2023-10-20T03:31:54.809 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:31:54.844 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-rbd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:31:54.903 INFO:teuthology.orchestra.run.smithi033.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6703-g8a5df55d-1jammy [120 kB] 2023-10-20T03:31:54.906 INFO:teuthology.orchestra.run.smithi033.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6703-g8a5df55d-1jammy [38.3 kB] 2023-10-20T03:31:54.907 INFO:teuthology.orchestra.run.smithi033.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6703-g8a5df55d-1jammy [12.3 MB] 2023-10-20T03:31:55.197 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-20T03:31:55.223 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-20T03:31:55.273 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-20T03:31:55.372 INFO:teuthology.orchestra.run.smithi064.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6703-g8a5df55d-1jammy [120 kB] 2023-10-20T03:31:55.374 INFO:teuthology.orchestra.run.smithi064.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6703-g8a5df55d-1jammy [38.3 kB] 2023-10-20T03:31:55.375 INFO:teuthology.orchestra.run.smithi064.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6703-g8a5df55d-1jammy [12.3 MB] 2023-10-20T03:31:55.592 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package librgw2. 2023-10-20T03:31:55.618 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../019-librgw2_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:31:55.658 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking librgw2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:31:56.162 INFO:teuthology.orchestra.run.smithi033.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6703-g8a5df55d-1jammy [91.4 kB] 2023-10-20T03:31:56.185 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-rgw. 2023-10-20T03:31:56.211 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:31:56.252 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-rgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:31:56.569 INFO:teuthology.orchestra.run.smithi064.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6703-g8a5df55d-1jammy [91.4 kB] 2023-10-20T03:31:56.605 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-20T03:31:56.631 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-20T03:31:56.663 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-20T03:31:56.749 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 184 MB in 20s (9,220 kB/s) 2023-10-20T03:31:56.933 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-20T03:31:56.966 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libradosstriper1. 2023-10-20T03:31:56.992 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:31:57.032 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libradosstriper1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:31:57.037 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165984 files and directories currently installed.) 2023-10-20T03:31:57.042 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-20T03:31:57.132 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-20T03:31:57.144 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 184 MB in 21s (8,952 kB/s) 2023-10-20T03:31:57.283 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-20T03:31:57.349 INFO:teuthology.orchestra.run.smithi040.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6703-g8a5df55d-1jammy [120 kB] 2023-10-20T03:31:57.351 INFO:teuthology.orchestra.run.smithi040.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6703-g8a5df55d-1jammy [38.3 kB] 2023-10-20T03:31:57.353 INFO:teuthology.orchestra.run.smithi040.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6703-g8a5df55d-1jammy [12.3 MB] 2023-10-20T03:31:57.383 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165984 files and directories currently installed.) 2023-10-20T03:31:57.389 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-20T03:31:57.452 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-20T03:31:57.453 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-common. 2023-10-20T03:31:57.473 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-20T03:31:57.478 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-20T03:31:57.479 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:31:57.510 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-20T03:31:57.519 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:31:57.689 INFO:teuthology.orchestra.run.smithi040.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-10-11-0812/8a5df55d22d3638066184c241442ff00fdc0ebdd/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6703-g8a5df55d-1jammy [91.4 kB] 2023-10-20T03:31:57.817 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-20T03:31:57.844 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-20T03:31:57.847 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-20T03:31:57.874 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-20T03:31:57.884 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-20T03:31:57.914 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-20T03:31:58.229 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-20T03:31:58.250 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-20T03:31:58.256 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-20T03:31:58.266 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 184 MB in 22s (8,534 kB/s) 2023-10-20T03:31:58.276 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-20T03:31:58.296 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-20T03:31:58.325 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-20T03:31:58.409 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-20T03:31:58.510 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165984 files and directories currently installed.) 2023-10-20T03:31:58.516 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-20T03:31:58.600 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-20T03:31:58.615 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-20T03:31:58.642 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-20T03:31:58.674 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-20T03:31:58.775 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-base. 2023-10-20T03:31:58.801 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:31:58.867 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-base (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:31:58.870 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-20T03:31:58.897 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T03:31:58.953 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:31:58.955 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-20T03:31:58.982 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-20T03:31:59.021 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-20T03:31:59.136 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-20T03:31:59.162 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T03:31:59.211 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:31:59.357 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-20T03:31:59.358 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-20T03:31:59.384 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T03:31:59.385 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-20T03:31:59.423 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:31:59.424 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-20T03:31:59.613 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-20T03:31:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T03:31:59.646 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-20T03:31:59.672 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-20T03:31:59.680 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:31:59.720 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-20T03:31:59.770 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-20T03:31:59.796 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-20T03:31:59.810 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-20T03:31:59.836 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-20T03:31:59.837 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T03:31:59.885 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:31:59.975 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-20T03:32:00.002 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T03:32:00.041 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:32:00.090 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-cheroot. 2023-10-20T03:32:00.116 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-20T03:32:00.165 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-20T03:32:00.198 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-20T03:32:00.226 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T03:32:00.273 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:32:00.297 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-20T03:32:00.323 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-20T03:32:00.363 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-20T03:32:00.395 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-20T03:32:00.422 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-20T03:32:00.461 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-20T03:32:00.476 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-20T03:32:00.493 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-20T03:32:00.526 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-20T03:32:00.691 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librados2. 2023-10-20T03:32:00.718 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../008-librados2_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:00.758 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librados2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:00.789 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librados2. 2023-10-20T03:32:00.801 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-20T03:32:00.816 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../008-librados2_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:00.821 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-20T03:32:00.828 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T03:32:00.847 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-20T03:32:00.856 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librados2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:00.868 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:32:00.887 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-20T03:32:01.171 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-20T03:32:01.190 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-20T03:32:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-20T03:32:01.216 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-20T03:32:01.237 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:32:01.244 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librbd1. 2023-10-20T03:32:01.256 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-20T03:32:01.267 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librbd1. 2023-10-20T03:32:01.270 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../009-librbd1_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:01.294 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../009-librbd1_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:01.319 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librbd1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:01.325 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librbd1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:01.549 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-20T03:32:01.568 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-tempora. 2023-10-20T03:32:01.576 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-20T03:32:01.594 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-20T03:32:01.632 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-20T03:32:01.634 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-20T03:32:01.797 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libcephfs2. 2023-10-20T03:32:01.824 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:01.895 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-portend. 2023-10-20T03:32:01.905 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libcephfs2. 2023-10-20T03:32:01.910 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-20T03:32:01.931 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:01.943 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librados2. 2023-10-20T03:32:01.945 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-20T03:32:01.971 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../008-librados2_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:01.978 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libcephfs2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:01.997 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libcephfs2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:02.010 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librados2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:02.248 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-20T03:32:02.258 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rados. 2023-10-20T03:32:02.272 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rados. 2023-10-20T03:32:02.273 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:02.274 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-20T03:32:02.299 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:02.314 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-20T03:32:02.316 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rados (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:02.347 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rados (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:02.455 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librbd1. 2023-10-20T03:32:02.481 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../009-librbd1_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:02.521 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librbd1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:02.618 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-20T03:32:02.636 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-20T03:32:02.644 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-20T03:32:02.663 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:02.684 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-20T03:32:02.702 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-ceph-argparse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:02.733 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-20T03:32:02.760 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:02.808 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-ceph-argparse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:02.924 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libcephfs2. 2023-10-20T03:32:02.951 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:02.982 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libcephfs2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:03.005 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cephfs. 2023-10-20T03:32:03.020 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:03.038 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-natsort. 2023-10-20T03:32:03.061 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cephfs. 2023-10-20T03:32:03.064 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-20T03:32:03.076 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:03.080 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cephfs (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:03.104 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-20T03:32:03.119 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cephfs (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:03.418 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rados. 2023-10-20T03:32:03.430 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-20T03:32:03.433 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-20T03:32:03.434 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:03.457 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:03.459 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:03.477 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rados (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:03.496 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:03.499 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:03.624 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-logutils. 2023-10-20T03:32:03.651 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-20T03:32:03.690 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-20T03:32:03.796 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-20T03:32:03.823 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:03.827 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-20T03:32:03.843 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-20T03:32:03.863 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-ceph-argparse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:03.866 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-20T03:32:03.886 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-20T03:32:03.893 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-20T03:32:03.941 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-20T03:32:03.994 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-20T03:32:04.020 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-20T03:32:04.060 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-20T03:32:04.116 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cephfs. 2023-10-20T03:32:04.131 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:04.174 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cephfs (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:04.214 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-prettytable. 2023-10-20T03:32:04.240 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-20T03:32:04.280 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-20T03:32:04.313 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-mako. 2023-10-20T03:32:04.319 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-prettytable. 2023-10-20T03:32:04.339 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-20T03:32:04.346 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-20T03:32:04.379 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-20T03:32:04.394 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-20T03:32:04.454 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-20T03:32:04.478 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:04.510 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:04.600 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rbd. 2023-10-20T03:32:04.615 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:04.663 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rbd. 2023-10-20T03:32:04.680 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:04.691 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rbd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:04.721 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rbd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:04.733 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-20T03:32:04.759 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-20T03:32:04.799 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-20T03:32:04.813 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-20T03:32:04.840 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-20T03:32:04.879 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-20T03:32:05.053 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-20T03:32:05.066 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-20T03:32:05.079 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-20T03:32:05.094 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-20T03:32:05.110 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-20T03:32:05.119 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-20T03:32:05.137 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-20T03:32:05.141 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-20T03:32:05.177 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-20T03:32:05.191 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-prettytable. 2023-10-20T03:32:05.217 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-20T03:32:05.257 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-20T03:32:05.448 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librgw2. 2023-10-20T03:32:05.474 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../019-librgw2_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:05.480 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-webob. 2023-10-20T03:32:05.506 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-20T03:32:05.514 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librgw2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:05.526 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rbd. 2023-10-20T03:32:05.536 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librgw2. 2023-10-20T03:32:05.542 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:05.546 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-20T03:32:05.563 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../019-librgw2_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:05.585 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rbd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:05.627 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librgw2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:05.880 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-20T03:32:05.882 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-waitress. 2023-10-20T03:32:05.896 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-20T03:32:05.909 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-20T03:32:05.929 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-20T03:32:05.944 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-20T03:32:06.058 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rgw. 2023-10-20T03:32:06.085 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:06.133 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:06.180 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rgw. 2023-10-20T03:32:06.207 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:06.219 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-tempita. 2023-10-20T03:32:06.245 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-20T03:32:06.246 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:06.266 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librgw2. 2023-10-20T03:32:06.285 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-20T03:32:06.292 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../019-librgw2_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:06.332 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librgw2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:06.511 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-20T03:32:06.538 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-20T03:32:06.578 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-20T03:32:06.588 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-paste. 2023-10-20T03:32:06.591 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-20T03:32:06.614 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-20T03:32:06.618 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-20T03:32:06.655 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-20T03:32:06.658 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-20T03:32:06.881 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libradosstriper1. 2023-10-20T03:32:06.893 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rgw. 2023-10-20T03:32:06.896 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:06.909 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:06.936 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libradosstriper1. 2023-10-20T03:32:06.941 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-20T03:32:06.943 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libradosstriper1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:06.951 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:06.956 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-20T03:32:06.962 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:06.991 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-20T03:32:06.994 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libradosstriper1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:07.254 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-20T03:32:07.270 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-20T03:32:07.294 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-20T03:32:07.304 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-20T03:32:07.321 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-20T03:32:07.360 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-20T03:32:07.422 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-common. 2023-10-20T03:32:07.449 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:07.476 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-common. 2023-10-20T03:32:07.488 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:07.503 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:07.543 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:07.574 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libradosstriper1. 2023-10-20T03:32:07.601 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:07.641 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libradosstriper1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:07.647 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-webtest. 2023-10-20T03:32:07.673 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-20T03:32:07.713 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-20T03:32:08.025 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-pecan. 2023-10-20T03:32:08.051 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-20T03:32:08.086 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-common. 2023-10-20T03:32:08.091 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-20T03:32:08.112 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:08.152 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:08.454 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-certifi. 2023-10-20T03:32:08.481 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-20T03:32:08.523 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-20T03:32:08.911 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-base. 2023-10-20T03:32:08.938 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:08.940 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-base. 2023-10-20T03:32:08.967 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:08.996 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-base (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:09.016 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-base (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:09.657 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-idna. 2023-10-20T03:32:09.673 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-20T03:32:09.715 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-20T03:32:10.018 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-urllib3. 2023-10-20T03:32:10.045 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-20T03:32:10.085 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-20T03:32:10.167 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-base. 2023-10-20T03:32:10.194 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:10.243 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-base (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:10.406 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-requests. 2023-10-20T03:32:10.432 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-20T03:32:10.471 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-20T03:32:10.549 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-20T03:32:10.553 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-20T03:32:10.576 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-20T03:32:10.580 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-20T03:32:10.616 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-20T03:32:10.619 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-20T03:32:10.758 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-20T03:32:10.784 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-20T03:32:10.824 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-20T03:32:10.864 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cheroot. 2023-10-20T03:32:10.880 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-20T03:32:10.914 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-20T03:32:10.918 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cheroot. 2023-10-20T03:32:10.945 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-20T03:32:10.985 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-20T03:32:10.988 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-20T03:32:11.017 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-20T03:32:11.054 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-20T03:32:11.118 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-20T03:32:11.134 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:11.177 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:11.201 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-20T03:32:11.227 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-20T03:32:11.267 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-20T03:32:11.313 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-20T03:32:11.340 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-20T03:32:11.357 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cheroot. 2023-10-20T03:32:11.379 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-20T03:32:11.384 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-20T03:32:11.423 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-20T03:32:11.515 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-20T03:32:11.530 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:11.564 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:11.570 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-20T03:32:11.597 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-20T03:32:11.637 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-20T03:32:11.649 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-20T03:32:11.665 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-20T03:32:11.699 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-20T03:32:11.718 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-20T03:32:11.734 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-20T03:32:11.768 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-20T03:32:11.859 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-openssl. 2023-10-20T03:32:11.886 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-20T03:32:11.926 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-20T03:32:11.931 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-20T03:32:11.958 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-20T03:32:11.977 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-20T03:32:11.998 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-20T03:32:12.004 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-20T03:32:12.046 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-20T03:32:12.046 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-20T03:32:12.073 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-20T03:32:12.113 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-20T03:32:12.270 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-mgr. 2023-10-20T03:32:12.280 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-tempora. 2023-10-20T03:32:12.298 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:12.306 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-20T03:32:12.338 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-20T03:32:12.354 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-mgr (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:12.416 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-20T03:32:12.442 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-tempora. 2023-10-20T03:32:12.443 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-20T03:32:12.458 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-20T03:32:12.482 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-20T03:32:12.492 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-20T03:32:12.649 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-portend. 2023-10-20T03:32:12.676 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-20T03:32:12.715 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-20T03:32:12.735 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-tempora. 2023-10-20T03:32:12.751 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-20T03:32:12.757 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-mon. 2023-10-20T03:32:12.784 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:12.785 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-20T03:32:12.787 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-portend. 2023-10-20T03:32:12.814 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-20T03:32:12.823 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-mon (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:12.854 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-20T03:32:13.052 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-20T03:32:13.079 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-20T03:32:13.088 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-portend. 2023-10-20T03:32:13.115 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-20T03:32:13.118 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-20T03:32:13.155 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-20T03:32:13.165 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-20T03:32:13.192 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-20T03:32:13.232 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-20T03:32:13.379 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-20T03:32:13.395 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-20T03:32:13.408 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-20T03:32:13.429 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-20T03:32:13.434 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-20T03:32:13.466 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-20T03:32:13.510 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-20T03:32:13.526 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-20T03:32:13.559 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-20T03:32:13.733 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-natsort. 2023-10-20T03:32:13.749 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-20T03:32:13.769 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-20T03:32:13.783 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-20T03:32:13.796 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-20T03:32:13.836 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-20T03:32:13.890 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-natsort. 2023-10-20T03:32:13.915 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-20T03:32:13.927 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-osd. 2023-10-20T03:32:13.954 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:13.963 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-20T03:32:13.985 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-osd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:14.086 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-logutils. 2023-10-20T03:32:14.113 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-20T03:32:14.152 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-20T03:32:14.173 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-natsort. 2023-10-20T03:32:14.188 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-20T03:32:14.226 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-20T03:32:14.233 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-logutils. 2023-10-20T03:32:14.260 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-20T03:32:14.301 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-20T03:32:14.455 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-20T03:32:14.482 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-20T03:32:14.517 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-logutils. 2023-10-20T03:32:14.521 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-20T03:32:14.544 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-20T03:32:14.578 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-20T03:32:14.584 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-20T03:32:14.605 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-20T03:32:14.644 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-20T03:32:14.792 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-mako. 2023-10-20T03:32:14.818 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-20T03:32:14.849 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-20T03:32:14.887 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-20T03:32:14.914 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-20T03:32:14.953 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-20T03:32:14.955 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-mako. 2023-10-20T03:32:14.983 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-20T03:32:15.022 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-20T03:32:15.161 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-20T03:32:15.188 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-20T03:32:15.228 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-20T03:32:15.256 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-mako. 2023-10-20T03:32:15.283 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-20T03:32:15.323 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-20T03:32:15.326 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-20T03:32:15.352 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-20T03:32:15.392 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-20T03:32:15.530 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-20T03:32:15.557 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-20T03:32:15.597 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-20T03:32:15.676 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-20T03:32:15.695 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-20T03:32:15.703 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-20T03:32:15.722 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-20T03:32:15.734 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-20T03:32:15.762 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-20T03:32:15.838 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph. 2023-10-20T03:32:15.865 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../059-ceph_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:15.905 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:15.908 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-webob. 2023-10-20T03:32:15.935 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-20T03:32:15.975 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-20T03:32:16.012 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-20T03:32:16.031 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-webob. 2023-10-20T03:32:16.039 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-20T03:32:16.058 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-20T03:32:16.079 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-20T03:32:16.089 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-20T03:32:16.207 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-fuse. 2023-10-20T03:32:16.234 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:16.274 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:16.311 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-waitress. 2023-10-20T03:32:16.338 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-20T03:32:16.365 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-webob. 2023-10-20T03:32:16.381 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-20T03:32:16.392 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-20T03:32:16.431 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-20T03:32:16.468 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-waitress. 2023-10-20T03:32:16.495 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-20T03:32:16.538 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-20T03:32:16.652 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-mds. 2023-10-20T03:32:16.679 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:16.689 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-tempita. 2023-10-20T03:32:16.716 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-20T03:32:16.718 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-mds (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:16.755 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-20T03:32:16.793 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-waitress. 2023-10-20T03:32:16.812 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-tempita. 2023-10-20T03:32:16.820 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-20T03:32:16.839 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-20T03:32:16.863 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-20T03:32:16.870 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-20T03:32:16.994 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-paste. 2023-10-20T03:32:17.021 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-20T03:32:17.060 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-20T03:32:17.089 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package cephadm. 2023-10-20T03:32:17.115 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../062-cephadm_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:17.155 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:17.179 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-tempita. 2023-10-20T03:32:17.206 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-20T03:32:17.227 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-paste. 2023-10-20T03:32:17.246 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-20T03:32:17.254 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-20T03:32:17.357 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-20T03:32:17.388 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-20T03:32:17.404 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-20T03:32:17.446 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-20T03:32:17.475 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-20T03:32:17.499 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-paste. 2023-10-20T03:32:17.501 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-20T03:32:17.526 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-20T03:32:17.541 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-20T03:32:17.557 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-20T03:32:17.685 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-20T03:32:17.701 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-20T03:32:17.732 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-20T03:32:17.743 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-20T03:32:17.760 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-20T03:32:17.799 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-20T03:32:17.869 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-20T03:32:17.869 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-20T03:32:17.896 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-20T03:32:17.896 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-20T03:32:17.927 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-20T03:32:17.935 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-20T03:32:18.004 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-20T03:32:18.031 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-20T03:32:18.069 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-webtest. 2023-10-20T03:32:18.071 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-20T03:32:18.096 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-20T03:32:18.127 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-20T03:32:18.205 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-20T03:32:18.232 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-20T03:32:18.271 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-20T03:32:18.341 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-webtest. 2023-10-20T03:32:18.368 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-20T03:32:18.396 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pecan. 2023-10-20T03:32:18.399 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-20T03:32:18.424 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-20T03:32:18.463 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-20T03:32:18.574 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-webtest. 2023-10-20T03:32:18.601 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-20T03:32:18.641 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-20T03:32:18.669 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pecan. 2023-10-20T03:32:18.696 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-20T03:32:18.735 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-20T03:32:18.810 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-certifi. 2023-10-20T03:32:18.837 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-20T03:32:18.876 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-20T03:32:18.952 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pecan. 2023-10-20T03:32:18.966 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-babel. 2023-10-20T03:32:18.980 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-20T03:32:18.993 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-20T03:32:19.032 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-20T03:32:19.082 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-certifi. 2023-10-20T03:32:19.109 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-20T03:32:19.149 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-20T03:32:19.187 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-idna. 2023-10-20T03:32:19.210 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-20T03:32:19.214 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-20T03:32:19.253 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-20T03:32:19.369 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-jinja2. 2023-10-20T03:32:19.395 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-20T03:32:19.435 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-20T03:32:19.468 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-idna. 2023-10-20T03:32:19.495 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-20T03:32:19.532 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-certifi. 2023-10-20T03:32:19.535 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-20T03:32:19.559 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-20T03:32:19.565 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-urllib3. 2023-10-20T03:32:19.590 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-20T03:32:19.592 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-20T03:32:19.631 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-20T03:32:19.780 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-20T03:32:19.811 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:19.846 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:19.847 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-urllib3. 2023-10-20T03:32:19.873 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-20T03:32:19.893 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-idna. 2023-10-20T03:32:19.913 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-20T03:32:19.920 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-20T03:32:19.951 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-requests. 2023-10-20T03:32:19.959 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-20T03:32:19.978 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-20T03:32:20.018 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-20T03:32:20.183 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-20T03:32:20.210 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-20T03:32:20.250 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-20T03:32:20.258 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-requests. 2023-10-20T03:32:20.263 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-urllib3. 2023-10-20T03:32:20.285 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-20T03:32:20.290 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-20T03:32:20.316 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-20T03:32:20.321 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-20T03:32:20.329 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-20T03:32:20.356 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-20T03:32:20.395 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-20T03:32:20.553 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-20T03:32:20.579 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-20T03:32:20.591 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-requests. 2023-10-20T03:32:20.618 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-20T03:32:20.619 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-20T03:32:20.627 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-20T03:32:20.655 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-20T03:32:20.658 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-20T03:32:20.690 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-20T03:32:20.694 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-20T03:32:20.706 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:20.748 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:20.889 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-routes. 2023-10-20T03:32:20.904 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-20T03:32:20.938 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-20T03:32:20.960 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-20T03:32:20.980 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-20T03:32:20.987 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-20T03:32:20.996 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:21.027 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-20T03:32:21.041 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:21.095 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-20T03:32:21.122 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:21.153 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:21.259 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-20T03:32:21.286 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:21.317 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:21.338 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-20T03:32:21.366 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:21.396 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-20T03:32:21.405 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:21.423 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:21.462 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:21.472 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-openssl. 2023-10-20T03:32:21.499 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-20T03:32:21.539 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-20T03:32:21.751 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-20T03:32:21.762 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-20T03:32:21.767 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:21.781 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-openssl. 2023-10-20T03:32:21.789 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-20T03:32:21.801 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:21.808 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-20T03:32:21.829 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-20T03:32:21.848 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-20T03:32:21.883 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr. 2023-10-20T03:32:21.910 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:21.950 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:22.113 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-openssl. 2023-10-20T03:32:22.139 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-20T03:32:22.188 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-20T03:32:22.193 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr. 2023-10-20T03:32:22.220 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:22.259 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:22.312 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mon. 2023-10-20T03:32:22.339 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:22.349 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-joblib. 2023-10-20T03:32:22.376 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-20T03:32:22.378 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mon (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:22.415 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-20T03:32:22.541 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr. 2023-10-20T03:32:22.568 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:22.607 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:22.621 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mon. 2023-10-20T03:32:22.637 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:22.671 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mon (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:22.794 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-20T03:32:22.821 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-20T03:32:22.861 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-20T03:32:22.961 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mon. 2023-10-20T03:32:22.987 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:23.018 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mon (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:23.180 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-sklearn. 2023-10-20T03:32:23.207 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-20T03:32:23.246 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-20T03:32:23.281 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-osd. 2023-10-20T03:32:23.309 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:23.348 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-osd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:23.525 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-osd. 2023-10-20T03:32:23.551 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:23.590 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-osd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:23.931 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-osd. 2023-10-20T03:32:23.958 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:23.997 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-osd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:24.105 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-20T03:32:24.133 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:24.172 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:25.194 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-cachetools. 2023-10-20T03:32:25.221 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-20T03:32:25.243 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph. 2023-10-20T03:32:25.266 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-20T03:32:25.270 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../059-ceph_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:25.309 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:25.511 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph. 2023-10-20T03:32:25.527 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../059-ceph_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:25.562 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-fuse. 2023-10-20T03:32:25.589 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:25.619 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-20T03:32:25.620 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:25.646 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-20T03:32:25.677 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:25.694 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-20T03:32:25.972 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-fuse. 2023-10-20T03:32:25.990 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mds. 2023-10-20T03:32:25.992 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph. 2023-10-20T03:32:25.999 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:26.017 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:26.019 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../059-ceph_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:26.038 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:26.056 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mds (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:26.058 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:26.081 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-20T03:32:26.108 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-20T03:32:26.156 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-20T03:32:26.386 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-fuse. 2023-10-20T03:32:26.413 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:26.417 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mds. 2023-10-20T03:32:26.444 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:26.453 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:26.460 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package cephadm. 2023-10-20T03:32:26.483 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mds (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:26.487 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../062-cephadm_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:26.526 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:26.533 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-rsa. 2023-10-20T03:32:26.561 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-20T03:32:26.608 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-20T03:32:26.806 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mds. 2023-10-20T03:32:26.812 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-20T03:32:26.829 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-20T03:32:26.833 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:26.861 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package cephadm. 2023-10-20T03:32:26.862 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-20T03:32:26.873 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mds (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:26.888 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../062-cephadm_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:26.928 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:26.931 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-google-auth. 2023-10-20T03:32:26.958 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-20T03:32:26.997 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-20T03:32:27.207 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-20T03:32:27.209 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package cephadm. 2023-10-20T03:32:27.214 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-20T03:32:27.234 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-20T03:32:27.236 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../062-cephadm_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:27.241 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-20T03:32:27.273 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-20T03:32:27.276 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:27.281 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-20T03:32:27.342 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-20T03:32:27.369 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-20T03:32:27.409 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-20T03:32:27.587 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-20T03:32:27.614 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-20T03:32:27.654 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-20T03:32:27.742 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-20T03:32:27.769 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-20T03:32:27.789 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-websocket. 2023-10-20T03:32:27.808 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-20T03:32:27.816 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-20T03:32:27.847 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-20T03:32:27.948 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-20T03:32:27.964 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-20T03:32:28.007 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-20T03:32:28.142 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-20T03:32:28.169 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-20T03:32:28.208 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-20T03:32:28.404 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-babel. 2023-10-20T03:32:28.431 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-20T03:32:28.470 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-20T03:32:28.781 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jinja2. 2023-10-20T03:32:28.809 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-20T03:32:28.830 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-babel. 2023-10-20T03:32:28.839 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-20T03:32:28.846 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-20T03:32:28.880 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-20T03:32:28.881 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-20T03:32:28.897 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:28.939 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:28.962 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-babel. 2023-10-20T03:32:28.989 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-20T03:32:29.028 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-20T03:32:29.176 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-20T03:32:29.183 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jinja2. 2023-10-20T03:32:29.203 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:29.210 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-20T03:32:29.234 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-20T03:32:29.234 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:29.249 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-20T03:32:29.261 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-20T03:32:29.317 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-20T03:32:29.398 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jinja2. 2023-10-20T03:32:29.425 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-20T03:32:29.465 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-20T03:32:29.521 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-20T03:32:29.545 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-20T03:32:29.578 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-20T03:32:29.579 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-20T03:32:29.605 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:29.644 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:29.804 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-20T03:32:29.831 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-20T03:32:29.857 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-20T03:32:29.870 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-20T03:32:29.884 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-20T03:32:29.923 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-20T03:32:29.947 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-20T03:32:29.975 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-20T03:32:30.014 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-20T03:32:30.060 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-20T03:32:30.087 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:30.126 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:30.148 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package jq. 2023-10-20T03:32:30.164 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-20T03:32:30.206 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-20T03:32:30.235 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-routes. 2023-10-20T03:32:30.262 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-20T03:32:30.301 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-20T03:32:30.317 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-20T03:32:30.344 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-20T03:32:30.383 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-20T03:32:30.442 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package socat. 2023-10-20T03:32:30.458 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-20T03:32:30.480 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-20T03:32:30.500 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-20T03:32:30.508 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-20T03:32:30.547 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-20T03:32:30.647 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-20T03:32:30.674 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:30.686 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-routes. 2023-10-20T03:32:30.713 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-20T03:32:30.730 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:30.753 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-20T03:32:30.778 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package xmlstarlet. 2023-10-20T03:32:30.794 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-20T03:32:30.828 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-20T03:32:30.833 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-20T03:32:30.860 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-20T03:32:30.899 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-20T03:32:31.106 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-20T03:32:31.108 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-20T03:32:31.124 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-20T03:32:31.133 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:31.157 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-20T03:32:31.172 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:31.183 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-test. 2023-10-20T03:32:31.194 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-routes. 2023-10-20T03:32:31.210 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:31.222 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-20T03:32:31.249 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-test (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:31.260 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-20T03:32:31.559 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-20T03:32:31.575 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-20T03:32:31.589 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-20T03:32:31.617 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:31.617 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-20T03:32:31.656 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:31.854 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-joblib. 2023-10-20T03:32:31.881 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-20T03:32:31.920 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-20T03:32:32.043 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-20T03:32:32.070 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-20T03:32:32.109 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-20T03:32:32.146 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-joblib. 2023-10-20T03:32:32.173 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-20T03:32:32.212 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-20T03:32:32.248 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-20T03:32:32.264 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-20T03:32:32.298 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-20T03:32:32.549 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-20T03:32:32.565 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-20T03:32:32.576 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-sklearn. 2023-10-20T03:32:32.599 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-20T03:32:32.603 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-20T03:32:32.604 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-joblib. 2023-10-20T03:32:32.631 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-20T03:32:32.642 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-20T03:32:32.671 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-20T03:32:32.944 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-sklearn. 2023-10-20T03:32:32.971 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-20T03:32:33.016 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-20T03:32:33.019 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-20T03:32:33.044 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-20T03:32:33.083 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-20T03:32:33.310 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-20T03:32:33.338 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:33.376 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:33.377 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-sklearn. 2023-10-20T03:32:33.405 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-20T03:32:33.444 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-20T03:32:33.470 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ceph-volume. 2023-10-20T03:32:33.498 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:33.528 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ceph-volume (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:33.711 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-20T03:32:33.739 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:33.777 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:33.807 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-20T03:32:33.823 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:33.857 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libcephfs-dev (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:34.136 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-20T03:32:34.164 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:34.202 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:34.227 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package nvme-cli. 2023-10-20T03:32:34.254 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-20T03:32:34.293 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-20T03:32:34.404 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cachetools. 2023-10-20T03:32:34.432 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-20T03:32:34.471 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-20T03:32:34.632 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-20T03:32:34.648 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-20T03:32:34.690 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-20T03:32:34.732 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-20T03:32:34.739 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cachetools. 2023-10-20T03:32:34.760 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-20T03:32:34.766 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-20T03:32:34.790 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-20T03:32:34.805 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-20T03:32:35.077 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-20T03:32:35.104 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-20T03:32:35.108 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-20T03:32:35.119 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-20T03:32:35.135 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-20T03:32:35.135 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-20T03:32:35.143 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-20T03:32:35.156 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cachetools. 2023-10-20T03:32:35.177 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-20T03:32:35.183 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-20T03:32:35.183 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-20T03:32:35.223 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-20T03:32:35.430 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rsa. 2023-10-20T03:32:35.458 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-20T03:32:35.488 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-pastescript. 2023-10-20T03:32:35.496 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-20T03:32:35.511 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-20T03:32:35.516 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-20T03:32:35.525 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-20T03:32:35.538 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-20T03:32:35.553 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-20T03:32:35.555 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-20T03:32:35.577 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-20T03:32:35.592 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-20T03:32:35.816 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-google-auth. 2023-10-20T03:32:35.844 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-20T03:32:35.875 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-pluggy. 2023-10-20T03:32:35.882 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-20T03:32:35.903 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-20T03:32:35.914 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rsa. 2023-10-20T03:32:35.929 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-20T03:32:35.941 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-20T03:32:35.943 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-20T03:32:35.957 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-20T03:32:35.980 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-20T03:32:35.995 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-20T03:32:36.203 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-psutil. 2023-10-20T03:32:36.211 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-20T03:32:36.231 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-20T03:32:36.238 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-20T03:32:36.259 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-google-auth. 2023-10-20T03:32:36.269 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-20T03:32:36.277 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-20T03:32:36.287 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-20T03:32:36.290 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rsa. 2023-10-20T03:32:36.317 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-20T03:32:36.318 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-20T03:32:36.356 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-20T03:32:36.598 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-py. 2023-10-20T03:32:36.611 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-20T03:32:36.622 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-20T03:32:36.639 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-20T03:32:36.640 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-websocket. 2023-10-20T03:32:36.656 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-20T03:32:36.668 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-20T03:32:36.677 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-google-auth. 2023-10-20T03:32:36.678 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-20T03:32:36.698 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-20T03:32:36.704 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-20T03:32:36.743 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-20T03:32:36.976 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-pygments. 2023-10-20T03:32:37.003 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-20T03:32:37.010 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-20T03:32:37.038 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-20T03:32:37.042 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-20T03:32:37.046 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-20T03:32:37.074 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-20T03:32:37.075 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-websocket. 2023-10-20T03:32:37.076 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-20T03:32:37.102 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-20T03:32:37.113 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-20T03:32:37.141 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-20T03:32:37.503 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-20T03:32:37.513 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-20T03:32:37.520 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-websocket. 2023-10-20T03:32:37.530 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-20T03:32:37.541 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-20T03:32:37.547 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-20T03:32:37.571 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-20T03:32:37.577 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-20T03:32:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-20T03:32:37.774 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-20T03:32:37.802 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:37.874 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-toml. 2023-10-20T03:32:37.890 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:37.898 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-20T03:32:37.902 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-20T03:32:37.926 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-20T03:32:37.941 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-20T03:32:37.964 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-20T03:32:38.241 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-20T03:32:38.243 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-pytest. 2023-10-20T03:32:38.258 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:38.271 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-20T03:32:38.284 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-20T03:32:38.299 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:38.310 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-20T03:32:38.312 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-20T03:32:38.359 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-20T03:32:38.619 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-20T03:32:38.636 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-20T03:32:38.646 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-20T03:32:38.653 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:38.664 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package python3-simplejson. 2023-10-20T03:32:38.685 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-20T03:32:38.691 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-20T03:32:38.695 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:38.696 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-20T03:32:38.724 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-20T03:32:38.730 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-20T03:32:38.762 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-20T03:32:39.000 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-20T03:32:39.022 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-20T03:32:39.023 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package jq. 2023-10-20T03:32:39.027 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-20T03:32:39.040 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-20T03:32:39.047 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-20T03:32:39.050 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-20T03:32:39.066 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-20T03:32:39.074 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-20T03:32:39.089 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-20T03:32:39.090 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-20T03:32:39.142 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-20T03:32:39.399 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package jq. 2023-10-20T03:32:39.409 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package socat. 2023-10-20T03:32:39.415 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-20T03:32:39.426 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-20T03:32:39.426 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-20T03:32:39.449 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-20T03:32:39.453 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-20T03:32:39.475 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-20T03:32:39.483 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-20T03:32:39.718 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package socat. 2023-10-20T03:32:39.735 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-20T03:32:39.736 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package jq. 2023-10-20T03:32:39.752 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-20T03:32:39.776 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-20T03:32:39.787 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package xmlstarlet. 2023-10-20T03:32:39.794 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-20T03:32:39.815 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-20T03:32:39.854 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-20T03:32:40.071 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package xmlstarlet. 2023-10-20T03:32:40.079 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package radosgw. 2023-10-20T03:32:40.087 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-20T03:32:40.107 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../107-radosgw_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:40.129 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-20T03:32:40.145 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking radosgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:40.216 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-test. 2023-10-20T03:32:40.245 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:40.247 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package socat. 2023-10-20T03:32:40.275 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-20T03:32:40.282 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-test (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:40.314 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-20T03:32:40.467 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-test. 2023-10-20T03:32:40.495 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:40.533 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-test (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:40.609 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package xmlstarlet. 2023-10-20T03:32:40.625 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-20T03:32:40.667 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-20T03:32:40.923 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package rbd-fuse. 2023-10-20T03:32:40.951 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:40.990 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking rbd-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:41.021 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-test. 2023-10-20T03:32:41.049 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:41.087 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-test (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:41.293 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-20T03:32:41.602 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-20T03:32:41.704 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-20T03:32:41.940 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-20T03:32:42.309 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-20T03:32:42.753 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-volume. 2023-10-20T03:32:42.782 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:42.788 INFO:teuthology.orchestra.run.smithi170.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-20T03:32:42.828 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-volume (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:42.898 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:43.037 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-volume. 2023-10-20T03:32:43.066 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:43.104 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-volume (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:43.182 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-20T03:32:43.210 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:43.257 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libcephfs-dev (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:43.433 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-20T03:32:43.449 INFO:teuthology.orchestra.run.smithi170.stdout:Adding system user cephadm....done 2023-10-20T03:32:43.461 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:43.499 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libcephfs-dev (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:43.542 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-volume. 2023-10-20T03:32:43.570 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6703-g8a5df55d-1jammy_all.deb ... 2023-10-20T03:32:43.571 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-20T03:32:43.608 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-volume (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:43.660 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package nvme-cli. 2023-10-20T03:32:43.688 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-20T03:32:43.726 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-20T03:32:43.811 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package nvme-cli. 2023-10-20T03:32:43.822 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-20T03:32:43.839 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-20T03:32:43.869 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-20T03:32:43.921 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-20T03:32:43.937 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:43.978 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libcephfs-dev (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:44.082 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-20T03:32:44.090 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-20T03:32:44.098 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-20T03:32:44.140 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-20T03:32:44.191 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-20T03:32:44.207 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-20T03:32:44.217 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-20T03:32:44.249 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-20T03:32:44.323 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package nvme-cli. 2023-10-20T03:32:44.349 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-20T03:32:44.390 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-20T03:32:44.476 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-20T03:32:44.560 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-20T03:32:44.588 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-20T03:32:44.610 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-20T03:32:44.626 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-20T03:32:44.638 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-20T03:32:44.669 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-20T03:32:44.737 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-20T03:32:44.744 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-20T03:32:44.753 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-20T03:32:44.787 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-20T03:32:44.870 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-20T03:32:44.904 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pastescript. 2023-10-20T03:32:44.920 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-20T03:32:44.954 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-20T03:32:44.971 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pastescript. 2023-10-20T03:32:44.999 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-20T03:32:45.038 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-20T03:32:45.163 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-20T03:32:45.198 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-20T03:32:45.226 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-20T03:32:45.232 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pluggy. 2023-10-20T03:32:45.260 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-20T03:32:45.264 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-20T03:32:45.299 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-20T03:32:45.366 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pluggy. 2023-10-20T03:32:45.394 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-20T03:32:45.432 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-20T03:32:45.515 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-20T03:32:45.526 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pastescript. 2023-10-20T03:32:45.554 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-20T03:32:45.592 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-20T03:32:45.610 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-psutil. 2023-10-20T03:32:45.638 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-20T03:32:45.676 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-20T03:32:45.727 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-psutil. 2023-10-20T03:32:45.755 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-20T03:32:45.774 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-20T03:32:45.794 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-20T03:32:45.921 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pluggy. 2023-10-20T03:32:45.949 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-20T03:32:45.987 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-20T03:32:45.996 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-py. 2023-10-20T03:32:46.024 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-20T03:32:46.059 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-20T03:32:46.062 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-20T03:32:46.113 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-py. 2023-10-20T03:32:46.141 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-20T03:32:46.180 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-20T03:32:46.299 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-psutil. 2023-10-20T03:32:46.319 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-20T03:32:46.327 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-20T03:32:46.365 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-20T03:32:46.399 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pygments. 2023-10-20T03:32:46.427 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-20T03:32:46.465 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-20T03:32:46.499 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pygments. 2023-10-20T03:32:46.528 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-20T03:32:46.566 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-20T03:32:46.578 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-20T03:32:46.685 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-py. 2023-10-20T03:32:46.713 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-20T03:32:46.751 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-20T03:32:46.812 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-ceph-argparse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:46.953 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-20T03:32:46.982 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-20T03:32:47.020 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-20T03:32:47.029 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-20T03:32:47.057 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-20T03:32:47.072 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-20T03:32:47.081 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pygments. 2023-10-20T03:32:47.095 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-20T03:32:47.108 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-20T03:32:47.146 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-20T03:32:47.198 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-20T03:32:47.272 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-toml. 2023-10-20T03:32:47.301 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-20T03:32:47.339 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-20T03:32:47.439 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-toml. 2023-10-20T03:32:47.449 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-20T03:32:47.467 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-20T03:32:47.505 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-20T03:32:47.575 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-20T03:32:47.634 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-20T03:32:47.641 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pytest. 2023-10-20T03:32:47.662 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-20T03:32:47.670 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-20T03:32:47.708 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-20T03:32:47.725 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-20T03:32:47.792 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pytest. 2023-10-20T03:32:47.820 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-20T03:32:47.858 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-20T03:32:47.934 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-20T03:32:48.020 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-simplejson. 2023-10-20T03:32:48.029 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-toml. 2023-10-20T03:32:48.036 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-20T03:32:48.056 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-20T03:32:48.060 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-20T03:32:48.069 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-20T03:32:48.094 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-20T03:32:48.289 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-20T03:32:48.305 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-20T03:32:48.329 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-20T03:32:48.347 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pytest. 2023-10-20T03:32:48.347 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-20T03:32:48.363 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-20T03:32:48.405 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-20T03:32:48.453 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-simplejson. 2023-10-20T03:32:48.481 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-20T03:32:48.520 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-20T03:32:48.571 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-20T03:32:48.734 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-simplejson. 2023-10-20T03:32:48.761 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-20T03:32:48.800 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-20T03:32:48.814 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-20T03:32:48.831 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-20T03:32:48.872 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-20T03:32:48.881 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-20T03:32:49.086 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-20T03:32:49.093 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package radosgw. 2023-10-20T03:32:49.102 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-20T03:32:49.121 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../107-radosgw_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:49.144 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-20T03:32:49.150 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking radosgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:49.524 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-20T03:32:49.551 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package radosgw. 2023-10-20T03:32:49.580 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../107-radosgw_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:49.618 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking radosgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:49.800 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-20T03:32:49.856 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package radosgw. 2023-10-20T03:32:49.885 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../107-radosgw_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:49.922 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking radosgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:49.926 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-20T03:32:49.929 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package rbd-fuse. 2023-10-20T03:32:49.957 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:49.995 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking rbd-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:50.052 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-20T03:32:50.316 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-20T03:32:50.362 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package rbd-fuse. 2023-10-20T03:32:50.363 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-20T03:32:50.379 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:50.420 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking rbd-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:50.575 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-20T03:32:50.701 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-20T03:32:50.724 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-20T03:32:50.730 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-20T03:32:50.818 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package rbd-fuse. 2023-10-20T03:32:50.834 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6703-g8a5df55d-1jammy_amd64.deb ... 2023-10-20T03:32:50.875 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking rbd-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:50.970 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-20T03:32:50.982 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-20T03:32:51.067 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-20T03:32:51.155 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-20T03:32:51.193 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-20T03:32:51.297 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-20T03:32:51.341 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-20T03:32:51.447 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-20T03:32:51.456 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-20T03:32:51.573 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-20T03:32:51.634 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-20T03:32:51.758 INFO:teuthology.orchestra.run.smithi064.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-20T03:32:51.806 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-20T03:32:51.843 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-20T03:32:51.844 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:51.977 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-20T03:32:52.206 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-20T03:32:52.237 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-20T03:32:52.295 INFO:teuthology.orchestra.run.smithi033.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-20T03:32:52.354 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:52.396 INFO:teuthology.orchestra.run.smithi064.stdout:Adding system user cephadm....done 2023-10-20T03:32:52.411 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:52.559 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-20T03:32:52.647 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-idna (3.3-1) ... 2023-10-20T03:32:52.663 INFO:teuthology.orchestra.run.smithi040.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-20T03:32:52.734 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:52.852 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-20T03:32:52.954 INFO:teuthology.orchestra.run.smithi033.stdout:Adding system user cephadm....done 2023-10-20T03:32:52.965 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-20T03:32:53.101 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-20T03:32:53.112 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-20T03:32:53.238 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-20T03:32:53.269 INFO:teuthology.orchestra.run.smithi040.stdout:Adding system user cephadm....done 2023-10-20T03:32:53.381 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-20T03:32:53.402 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-20T03:32:53.480 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-20T03:32:53.667 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-20T03:32:53.678 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-20T03:32:53.732 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-20T03:32:53.796 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-20T03:32:53.849 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-20T03:32:53.918 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-20T03:32:54.044 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-20T03:32:54.047 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-20T03:32:54.154 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:32:54.192 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-20T03:32:54.289 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-20T03:32:54.303 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-20T03:32:54.306 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-20T03:32:54.433 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-20T03:32:54.556 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-20T03:32:54.569 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-20T03:32:54.571 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-20T03:32:54.672 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-20T03:32:54.682 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-20T03:32:54.768 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-20T03:32:54.808 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-20T03:32:54.812 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-20T03:32:54.974 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-20T03:32:55.080 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-20T03:32:55.102 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-20T03:32:55.161 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-20T03:32:55.350 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-20T03:32:55.389 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-20T03:32:55.437 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-20T03:32:55.620 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-20T03:32:55.626 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-20T03:32:55.649 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-20T03:32:55.705 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-20T03:32:55.854 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:32:55.900 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-ceph-argparse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:55.902 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-20T03:32:55.964 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-20T03:32:55.980 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-20T03:32:56.106 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-20T03:32:56.153 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-20T03:32:56.176 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-20T03:32:56.232 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-20T03:32:56.302 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-20T03:32:56.407 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-20T03:32:56.421 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-20T03:32:56.492 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-ceph-argparse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:56.561 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-20T03:32:56.659 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-20T03:32:56.664 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-ceph-argparse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:32:56.687 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-20T03:32:56.759 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-20T03:32:56.910 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-20T03:32:56.918 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-20T03:32:56.923 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-20T03:32:57.050 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-20T03:32:57.063 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-20T03:32:57.161 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-20T03:32:57.189 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-20T03:32:57.195 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-20T03:32:57.301 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-20T03:32:57.304 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-20T03:32:57.402 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-20T03:32:57.404 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-20T03:32:57.449 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-20T03:32:57.655 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-20T03:32:57.655 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-20T03:32:57.716 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-20T03:32:57.769 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-20T03:32:57.781 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-20T03:32:57.781 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-20T03:32:57.903 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-20T03:32:58.043 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-20T03:32:58.057 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-20T03:32:58.065 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-20T03:32:58.180 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-20T03:32:58.191 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-20T03:32:58.333 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-20T03:32:58.448 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-20T03:32:58.484 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-20T03:32:58.651 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-20T03:32:58.738 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-20T03:32:58.774 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-20T03:32:58.802 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-20T03:32:58.997 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-20T03:32:59.054 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-20T03:32:59.098 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-20T03:32:59.216 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-20T03:32:59.387 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-20T03:32:59.409 INFO:teuthology.orchestra.run.smithi170.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-20T03:32:59.459 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-20T03:32:59.525 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-20T03:32:59.646 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-20T03:32:59.665 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-20T03:32:59.710 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-20T03:32:59.747 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-20T03:32:59.791 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-20T03:32:59.836 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-20T03:32:59.864 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-20T03:32:59.960 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-20T03:32:59.962 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-20T03:33:00.174 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-20T03:33:00.184 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-20T03:33:00.227 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-20T03:33:00.264 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-20T03:33:00.443 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-20T03:33:00.575 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-20T03:33:00.612 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-20T03:33:00.665 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-20T03:33:00.678 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:33:00.851 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-20T03:33:00.905 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-20T03:33:00.905 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-20T03:33:00.924 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-20T03:33:01.228 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-20T03:33:01.240 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-20T03:33:01.255 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-20T03:33:01.301 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-20T03:33:01.491 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-20T03:33:01.529 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-20T03:33:01.548 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-20T03:33:01.585 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-20T03:33:01.617 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:01.824 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-20T03:33:01.889 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-20T03:33:01.929 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-20T03:33:01.935 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-idna (3.3-1) ... 2023-10-20T03:33:01.950 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-20T03:33:02.148 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-20T03:33:02.179 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-20T03:33:02.243 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-20T03:33:02.244 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-20T03:33:02.274 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:02.289 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:02.511 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-20T03:33:02.590 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-idna (3.3-1) ... 2023-10-20T03:33:02.592 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-idna (3.3-1) ... 2023-10-20T03:33:02.754 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-20T03:33:02.918 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-20T03:33:02.925 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-20T03:33:03.130 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-20T03:33:03.431 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-20T03:33:03.475 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:33:03.601 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-20T03:33:03.808 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-20T03:33:03.876 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-20T03:33:03.935 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up librados2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:03.977 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-20T03:33:04.052 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up librgw2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:04.078 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-20T03:33:04.162 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:04.163 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:33:04.182 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:33:04.271 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libcephfs2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:04.298 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-20T03:33:04.342 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-20T03:33:04.380 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-20T03:33:04.397 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libradosstriper1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:04.557 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-20T03:33:04.609 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-20T03:33:04.626 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-20T03:33:04.639 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-20T03:33:04.676 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-20T03:33:04.776 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-20T03:33:04.777 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-20T03:33:04.882 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:33:04.885 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up librbd1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:04.928 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-20T03:33:04.986 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-mgr-modules-core (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:05.008 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-20T03:33:05.094 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-20T03:33:05.096 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:05.134 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-20T03:33:05.238 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-20T03:33:05.306 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-20T03:33:05.307 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-20T03:33:05.362 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-20T03:33:05.452 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-20T03:33:05.497 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-20T03:33:05.613 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:33:05.703 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-20T03:33:05.739 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-20T03:33:05.757 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:33:05.847 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libcephfs-dev (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:05.865 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-20T03:33:05.908 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-20T03:33:05.962 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-20T03:33:05.973 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-rados (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:06.059 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-20T03:33:06.100 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-20T03:33:06.230 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-20T03:33:06.384 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-20T03:33:06.402 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-20T03:33:06.490 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-20T03:33:06.635 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-20T03:33:06.678 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-20T03:33:06.774 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-20T03:33:06.911 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-20T03:33:06.916 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-20T03:33:06.945 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-20T03:33:07.170 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-20T03:33:07.205 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-20T03:33:07.217 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-20T03:33:07.271 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-rbd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:07.335 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-20T03:33:07.397 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up rbd-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:07.429 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-20T03:33:07.464 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-20T03:33:07.523 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-rgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:07.611 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-20T03:33:07.649 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up python3-cephfs (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:07.697 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-20T03:33:07.732 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-20T03:33:07.775 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:07.823 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-20T03:33:07.877 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-20T03:33:07.912 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-20T03:33:08.108 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-20T03:33:08.170 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-20T03:33:08.172 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-20T03:33:08.234 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-20T03:33:08.296 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-20T03:33:08.326 INFO:teuthology.orchestra.run.smithi170.stdout:Adding group ceph....done 2023-10-20T03:33:08.527 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-20T03:33:08.565 INFO:teuthology.orchestra.run.smithi064.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-20T03:33:08.588 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-20T03:33:08.694 INFO:teuthology.orchestra.run.smithi170.stdout:Adding system user ceph....done 2023-10-20T03:33:08.830 INFO:teuthology.orchestra.run.smithi170.stdout:Setting system user ceph properties....done 2023-10-20T03:33:08.833 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-20T03:33:08.836 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-20T03:33:08.845 INFO:teuthology.orchestra.run.smithi170.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-20T03:33:08.898 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-20T03:33:08.959 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-20T03:33:08.980 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-20T03:33:09.096 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-20T03:33:09.157 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-20T03:33:09.329 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-20T03:33:09.368 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-20T03:33:09.528 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-20T03:33:09.579 INFO:teuthology.orchestra.run.smithi033.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-20T03:33:09.620 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-20T03:33:09.782 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-20T03:33:09.852 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-20T03:33:09.877 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-test (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:09.879 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:33:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-20T03:33:09.988 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-20T03:33:09.995 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-20T03:33:10.011 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up radosgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:10.306 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-20T03:33:10.309 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-20T03:33:10.405 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-20T03:33:10.503 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-20T03:33:10.504 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-20T03:33:10.561 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-20T03:33:10.632 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-20T03:33:10.697 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-20T03:33:10.820 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:33:10.917 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-20T03:33:10.946 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-20T03:33:10.957 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-20T03:33:11.042 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-20T03:33:11.046 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-base (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:11.074 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-20T03:33:11.289 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-20T03:33:11.335 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-20T03:33:11.384 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-20T03:33:11.397 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-20T03:33:11.590 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-20T03:33:11.603 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-20T03:33:11.718 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-20T03:33:11.854 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-20T03:33:11.866 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-20T03:33:11.948 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-mds (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:11.992 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-20T03:33:11.994 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-20T03:33:12.120 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-20T03:33:12.182 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-20T03:33:12.183 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-20T03:33:12.222 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-20T03:33:12.260 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-20T03:33:12.405 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-20T03:33:12.507 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-20T03:33:12.536 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-20T03:33:12.681 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-20T03:33:12.741 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-mgr (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:12.804 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-20T03:33:12.875 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-20T03:33:12.949 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-20T03:33:12.963 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-20T03:33:12.963 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-20T03:33:13.001 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librados2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:13.102 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librgw2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:13.197 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-20T03:33:13.203 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:13.321 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libcephfs2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:13.342 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-20T03:33:13.447 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libradosstriper1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:13.481 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-20T03:33:13.501 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-osd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:13.667 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-20T03:33:13.668 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-20T03:33:13.815 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-20T03:33:13.816 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-20T03:33:13.825 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-20T03:33:13.935 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librados2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:13.968 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librbd1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.036 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librgw2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.047 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-20T03:33:14.094 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-modules-core (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.162 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.171 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librados2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.220 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.288 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libcephfs2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.314 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librgw2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.378 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.414 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libradosstriper1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.440 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-20T03:33:14.440 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-20T03:33:14.441 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.479 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.557 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libcephfs2 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.605 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-mon (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.625 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-20T03:33:14.792 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libradosstriper1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.816 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-20T03:33:14.816 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-20T03:33:14.910 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librbd1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:14.922 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libcephfs-dev (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:15.003 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-20T03:33:15.011 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-modules-core (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:15.039 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rados (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:15.137 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:15.166 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-20T03:33:15.280 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librbd1 (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:15.340 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-mgr-cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:15.357 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-20T03:33:15.357 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-20T03:33:15.406 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-modules-core (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:15.491 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:15.532 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:15.617 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-mgr-dashboard (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:15.743 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-20T03:33:15.744 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-20T03:33:15.759 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ceph-volume (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:15.897 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libcephfs-dev (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:16.023 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rados (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:16.096 INFO:teuthology.orchestra.run.smithi170.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-20T03:33:16.149 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-20T03:33:16.325 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libcephfs-dev (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:16.428 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rbd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:16.485 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rados (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:16.554 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up rbd-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:16.635 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-20T03:33:16.655 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:16.756 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cephfs (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:16.883 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:17.225 INFO:teuthology.orchestra.run.smithi170.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-20T03:33:17.425 INFO:teuthology.orchestra.run.smithi064.stdout:Adding group ceph....done 2023-10-20T03:33:17.470 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rbd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:17.596 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up rbd-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:17.690 INFO:teuthology.orchestra.run.smithi170.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-20T03:33:17.693 DEBUG:teuthology.parallel:result is None 2023-10-20T03:33:17.722 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:17.835 INFO:teuthology.orchestra.run.smithi064.stdout:Adding system user ceph....done 2023-10-20T03:33:17.848 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cephfs (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:17.923 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rbd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:17.974 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:18.012 INFO:teuthology.orchestra.run.smithi064.stdout:Setting system user ceph properties....done 2023-10-20T03:33:18.024 INFO:teuthology.orchestra.run.smithi064.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-20T03:33:18.040 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up rbd-fuse (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:18.141 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:18.165 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-20T03:33:18.251 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cephfs (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:18.377 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-common (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:18.510 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-20T03:33:18.517 INFO:teuthology.orchestra.run.smithi040.stdout:Adding group ceph....done 2023-10-20T03:33:18.835 INFO:teuthology.orchestra.run.smithi040.stdout:Adding system user ceph....done 2023-10-20T03:33:18.920 INFO:teuthology.orchestra.run.smithi033.stdout:Adding group ceph....done 2023-10-20T03:33:19.012 INFO:teuthology.orchestra.run.smithi040.stdout:Setting system user ceph properties....done 2023-10-20T03:33:19.032 INFO:teuthology.orchestra.run.smithi040.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-20T03:33:19.050 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-test (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:19.172 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-20T03:33:19.176 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up radosgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:19.372 INFO:teuthology.orchestra.run.smithi033.stdout:Adding system user ceph....done 2023-10-20T03:33:19.515 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-20T03:33:19.565 INFO:teuthology.orchestra.run.smithi033.stdout:Setting system user ceph properties....done 2023-10-20T03:33:19.583 INFO:teuthology.orchestra.run.smithi033.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-20T03:33:19.666 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-20T03:33:19.667 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-20T03:33:19.736 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-20T03:33:20.067 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-test (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:20.117 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-20T03:33:20.178 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-base (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:20.193 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up radosgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:20.527 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-20T03:33:20.637 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-test (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:20.684 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-20T03:33:20.684 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-20T03:33:20.763 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up radosgw (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:21.046 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mds (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:21.178 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-base (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:21.255 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-20T03:33:21.255 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-20T03:33:21.343 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-20T03:33:21.343 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-20T03:33:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-20T03:33:21.756 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-base (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:22.046 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mds (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:22.143 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-20T03:33:22.186 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:22.287 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-20T03:33:22.287 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-20T03:33:22.410 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-20T03:33:22.410 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-20T03:33:22.658 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mds (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:22.781 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:22.876 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-20T03:33:22.876 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-20T03:33:22.921 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-osd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:22.984 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-20T03:33:22.984 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-20T03:33:23.260 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-20T03:33:23.260 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-20T03:33:23.468 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:23.508 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-osd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:23.697 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-20T03:33:23.697 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-20T03:33:23.814 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:23.842 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-20T03:33:23.843 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-20T03:33:24.032 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:24.300 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mon (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:24.360 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:24.372 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-osd (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:24.469 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:24.522 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-20T03:33:24.522 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-20T03:33:24.636 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mon (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:24.694 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-20T03:33:24.694 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-20T03:33:24.852 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-20T03:33:24.852 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-20T03:33:25.025 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:25.176 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:25.216 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:25.302 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-dashboard (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:25.354 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:25.375 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:25.470 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-volume (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:25.505 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:25.602 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mon (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:25.631 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-dashboard (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:25.791 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-volume (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:25.814 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-20T03:33:25.815 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-20T03:33:25.848 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-20T03:33:26.136 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-20T03:33:26.377 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-cephadm (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:26.659 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:26.870 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-dashboard (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:26.974 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-20T03:33:27.106 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-volume (18.0.0-6703-g8a5df55d-1jammy) ... 2023-10-20T03:33:27.254 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-20T03:33:27.436 INFO:teuthology.orchestra.run.smithi064.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-20T03:33:27.439 DEBUG:teuthology.parallel:result is None 2023-10-20T03:33:27.525 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-20T03:33:27.725 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-20T03:33:27.728 DEBUG:teuthology.parallel:result is None 2023-10-20T03:33:28.217 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-20T03:33:28.680 INFO:teuthology.orchestra.run.smithi033.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-20T03:33:28.683 DEBUG:teuthology.parallel:result is None 2023-10-20T03:33:28.684 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8a5df55d22d3638066184c241442ff00fdc0ebdd 2023-10-20T03:33:28.904 DEBUG:teuthology.orchestra.run.smithi033:> dpkg-query -W -f '${Version}' ceph 2023-10-20T03:33:28.924 INFO:teuthology.orchestra.run.smithi033.stdout:18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:33:28.924 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:33:28.925 INFO:teuthology.task.install:The correct ceph version 18.0.0-6703-g8a5df55d-1jammy is installed. 2023-10-20T03:33:28.925 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8a5df55d22d3638066184c241442ff00fdc0ebdd 2023-10-20T03:33:29.141 DEBUG:teuthology.orchestra.run.smithi040:> dpkg-query -W -f '${Version}' ceph 2023-10-20T03:33:29.160 INFO:teuthology.orchestra.run.smithi040.stdout:18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:33:29.161 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:33:29.161 INFO:teuthology.task.install:The correct ceph version 18.0.0-6703-g8a5df55d-1jammy is installed. 2023-10-20T03:33:29.162 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8a5df55d22d3638066184c241442ff00fdc0ebdd 2023-10-20T03:33:29.375 DEBUG:teuthology.orchestra.run.smithi064:> dpkg-query -W -f '${Version}' ceph 2023-10-20T03:33:29.394 INFO:teuthology.orchestra.run.smithi064.stdout:18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:33:29.395 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:33:29.395 INFO:teuthology.task.install:The correct ceph version 18.0.0-6703-g8a5df55d-1jammy is installed. 2023-10-20T03:33:29.396 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8a5df55d22d3638066184c241442ff00fdc0ebdd 2023-10-20T03:33:29.534 DEBUG:teuthology.orchestra.run.smithi170:> dpkg-query -W -f '${Version}' ceph 2023-10-20T03:33:29.554 INFO:teuthology.orchestra.run.smithi170.stdout:18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:33:29.554 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6703-g8a5df55d-1jammy 2023-10-20T03:33:29.554 INFO:teuthology.task.install:The correct ceph version 18.0.0-6703-g8a5df55d-1jammy is installed. 2023-10-20T03:33:29.555 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-20T03:33:29.555 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:33:29.556 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-20T03:33:29.571 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:33:29.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-20T03:33:29.587 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:33:29.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-20T03:33:29.602 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:33:29.602 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-20T03:33:29.616 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-20T03:33:29.617 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:33:29.617 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2023-10-20T03:33:29.631 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-20T03:33:29.686 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:33:29.686 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2023-10-20T03:33:29.700 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-20T03:33:29.756 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:33:29.756 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2023-10-20T03:33:29.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-20T03:33:29.824 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:33:29.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2023-10-20T03:33:29.838 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-20T03:33:29.890 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-20T03:33:29.891 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:33:29.891 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-20T03:33:29.905 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-20T03:33:29.961 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:33:29.962 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-20T03:33:29.975 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-20T03:33:30.035 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:33:30.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-20T03:33:30.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-20T03:33:30.104 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:33:30.104 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-20T03:33:30.118 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-20T03:33:30.171 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-20T03:33:30.171 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:33:30.171 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2023-10-20T03:33:30.186 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-20T03:33:30.241 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:33:30.241 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2023-10-20T03:33:30.255 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-20T03:33:30.319 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:33:30.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2023-10-20T03:33:30.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-20T03:33:30.387 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:33:30.387 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2023-10-20T03:33:30.400 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-20T03:33:30.454 INFO:teuthology.run_tasks:Running task ceph... 2023-10-20T03:33:30.526 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-20T03:33:30.527 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /var/log/ceph 2023-10-20T03:33:30.529 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /var/log/ceph 2023-10-20T03:33:30.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2023-10-20T03:33:30.535 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /var/log/ceph 2023-10-20T03:33:30.547 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-20T03:33:30.547 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-20T03:33:30.583 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-20T03:33:30.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-20T03:33:30.590 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-20T03:33:30.603 INFO:tasks.ceph:Creating extra log directories... 2023-10-20T03:33:30.604 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-20T03:33:30.638 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-20T03:33:30.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-20T03:33:30.646 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-20T03:33:30.672 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-20T03:33:30.672 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-20T03:33:30.672 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476', 'branch': 'wip-yuri10-testing-2023-10-11-0812', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': None, 'first_in_suite': False, 'job_id': '7432476', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri10-testing-2023-10-11-0812'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '8a5df55d22d3638066184c241442ff00fdc0ebdd'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8a5df55d22d3638066184c241442ff00fdc0ebdd'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri10-testing-2023-10-11-0812', 'sha1': '8a5df55d22d3638066184c241442ff00fdc0ebdd'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 9009, 'sha1': '8a5df55d22d3638066184c241442ff00fdc0ebdd', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri10-testing-2023-10-11-0812', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8a5df55d22d3638066184c241442ff00fdc0ebdd', 'targets': {'smithi033.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKVOLgduYY8Px3G6GDW8+lkODtfqJDRiQdtvy3CJdcBka0VxAiKn06fsijKrvDtLUnWqN7FSFQQxQL+f/uE7L0A=', 'smithi040.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPl2EW4umy7LmnRQKJV/eZwxZaWNyUCOhFdnsV1xnxguPQ3cGwHK2gDCmNjhoifsNP69ssqA/e2XaUh7m5UejZo=', 'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEthpyLSta/xLwOp5moRS9tdSUTC7b2/1Ub3fauZiI4GC4QFVwvwI9RehZegOcyuYvXLyZr+5XHJP/D2yxtj0Lk=', 'smithi170.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCiptNh76uIuJqcjiE0raXNpFMJh2P8PY0fynAbDKQwiRO6c+CQE0wHU+LuDdd/46qgoWu9/nM+WCTFcmVQmPvQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '8a5df55d22d3638066184c241442ff00fdc0ebdd', 'cluster': 'ceph'}}, {'thrashosds': {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '202b180cb047e798fb131047314a862593f45403', 'timestamp': '2023-10-19_16:23:25', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-20T03:33:30.673 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-20T03:33:30.694 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-20T03:33:30.700 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-20T03:33:30.703 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-20T03:33:30.709 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/run/ceph 2023-10-20T03:33:30.743 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/run/ceph 2023-10-20T03:33:30.748 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2023-10-20T03:33:30.750 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m0777 -- /var/run/ceph 2023-10-20T03:33:30.766 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:33:30.767 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2023-10-20T03:33:30.801 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-20T03:33:30.802 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2023-10-20T03:33:30.850 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-20T03:33:30.851 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:30.851 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 881 Links: 1 2023-10-20T03:33:30.851 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:30.851 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-10-20 03:27:58.888598712 +0000 2023-10-20T03:33:30.851 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-10-20 03:27:58.540601821 +0000 2023-10-20T03:33:30.852 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-10-20 03:27:58.540601821 +0000 2023-10-20T03:33:30.852 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-10-20T03:33:30.852 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-20T03:33:30.904 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-10-20T03:33:30.905 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-10-20T03:33:30.905 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000441691 s, 1.2 MB/s 2023-10-20T03:33:30.906 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-20T03:33:30.955 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2023-10-20T03:33:31.001 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-20T03:33:31.002 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:31.002 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 905 Links: 1 2023-10-20T03:33:31.002 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:31.002 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-10-20 03:27:59.484593389 +0000 2023-10-20T03:33:31.002 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-10-20 03:27:59.132596534 +0000 2023-10-20T03:33:31.003 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-10-20 03:27:59.132596534 +0000 2023-10-20T03:33:31.003 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-10-20T03:33:31.003 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-20T03:33:31.056 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-10-20T03:33:31.056 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-10-20T03:33:31.056 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000306033 s, 1.7 MB/s 2023-10-20T03:33:31.057 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-20T03:33:31.107 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2023-10-20T03:33:31.154 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-20T03:33:31.154 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:31.154 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 921 Links: 1 2023-10-20T03:33:31.154 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:31.155 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-10-20 03:28:00.084588030 +0000 2023-10-20T03:33:31.155 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-10-20 03:27:59.724591246 +0000 2023-10-20T03:33:31.155 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-10-20 03:27:59.724591246 +0000 2023-10-20T03:33:31.155 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-10-20T03:33:31.156 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-20T03:33:31.208 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-10-20T03:33:31.208 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-10-20T03:33:31.209 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.00038059 s, 1.3 MB/s 2023-10-20T03:33:31.210 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-20T03:33:31.259 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2023-10-20T03:33:31.306 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-20T03:33:31.306 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:31.306 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 938 Links: 1 2023-10-20T03:33:31.307 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:31.307 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-10-20 03:28:00.656582920 +0000 2023-10-20T03:33:31.307 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-10-20 03:28:00.324585886 +0000 2023-10-20T03:33:31.307 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-10-20 03:28:00.324585886 +0000 2023-10-20T03:33:31.307 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-10-20T03:33:31.308 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-20T03:33:31.360 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-10-20T03:33:31.361 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-10-20T03:33:31.361 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.00040966 s, 1.2 MB/s 2023-10-20T03:33:31.362 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-20T03:33:31.411 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-10-20T03:33:31.411 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:33:31.411 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2023-10-20T03:33:31.417 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-20T03:33:31.418 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2023-10-20T03:33:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-20T03:33:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:31.464 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 883 Links: 1 2023-10-20T03:33:31.464 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:31.465 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-10-20 03:28:00.102552922 +0000 2023-10-20T03:33:31.465 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-10-20 03:27:59.726556276 +0000 2023-10-20T03:33:31.465 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-10-20 03:27:59.726556276 +0000 2023-10-20T03:33:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-10-20T03:33:31.466 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-20T03:33:31.518 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-10-20T03:33:31.518 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-10-20T03:33:31.518 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000445056 s, 1.2 MB/s 2023-10-20T03:33:31.519 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-20T03:33:31.570 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2023-10-20T03:33:31.615 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-20T03:33:31.615 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:31.616 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 899 Links: 1 2023-10-20T03:33:31.616 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:31.616 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-10-20 03:28:00.698547606 +0000 2023-10-20T03:33:31.616 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-10-20 03:28:00.334550852 +0000 2023-10-20T03:33:31.617 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-10-20 03:28:00.334550852 +0000 2023-10-20T03:33:31.617 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-10-20T03:33:31.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-20T03:33:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-10-20T03:33:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-10-20T03:33:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000410753 s, 1.2 MB/s 2023-10-20T03:33:31.671 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-20T03:33:31.721 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2023-10-20T03:33:31.767 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-20T03:33:31.767 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:31.768 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 922 Links: 1 2023-10-20T03:33:31.768 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:31.768 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-10-20 03:28:01.298542253 +0000 2023-10-20T03:33:31.768 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-10-20 03:28:00.962545251 +0000 2023-10-20T03:33:31.768 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-10-20 03:28:00.962545251 +0000 2023-10-20T03:33:31.769 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-10-20T03:33:31.769 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-20T03:33:31.822 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-10-20T03:33:31.822 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-10-20T03:33:31.822 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000336781 s, 1.5 MB/s 2023-10-20T03:33:31.823 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-20T03:33:31.872 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2023-10-20T03:33:31.919 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-20T03:33:31.919 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:31.919 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 940 Links: 1 2023-10-20T03:33:31.920 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:31.920 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-10-20 03:28:01.870537151 +0000 2023-10-20T03:33:31.920 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-10-20 03:28:01.538540112 +0000 2023-10-20T03:33:31.920 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-10-20 03:28:01.538540112 +0000 2023-10-20T03:33:31.920 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-10-20T03:33:31.921 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-20T03:33:31.974 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-10-20T03:33:31.974 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-10-20T03:33:31.975 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000387311 s, 1.3 MB/s 2023-10-20T03:33:31.976 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-20T03:33:32.029 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-10-20T03:33:32.029 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:33:32.030 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2023-10-20T03:33:32.035 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-20T03:33:32.036 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2023-10-20T03:33:32.084 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-20T03:33:32.085 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:32.085 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 889 Links: 1 2023-10-20T03:33:32.085 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:32.085 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-10-20 03:27:59.562412843 +0000 2023-10-20T03:33:32.085 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-10-20 03:27:59.190416166 +0000 2023-10-20T03:33:32.086 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-10-20 03:27:59.190416166 +0000 2023-10-20T03:33:32.086 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-10-20T03:33:32.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-20T03:33:32.139 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-10-20T03:33:32.139 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-10-20T03:33:32.139 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.00043671 s, 1.2 MB/s 2023-10-20T03:33:32.140 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-20T03:33:32.189 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2023-10-20T03:33:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-20T03:33:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:32.237 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 905 Links: 1 2023-10-20T03:33:32.237 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:32.237 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-10-20 03:28:00.150407591 +0000 2023-10-20T03:33:32.237 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-10-20 03:27:59.798410735 +0000 2023-10-20T03:33:32.237 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-10-20 03:27:59.798410735 +0000 2023-10-20T03:33:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-10-20T03:33:32.238 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-20T03:33:32.290 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-10-20T03:33:32.290 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-10-20T03:33:32.291 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000305224 s, 1.7 MB/s 2023-10-20T03:33:32.292 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-20T03:33:32.341 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2023-10-20T03:33:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-20T03:33:32.388 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:32.389 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 922 Links: 1 2023-10-20T03:33:32.389 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:32.389 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-10-20 03:28:00.762402124 +0000 2023-10-20T03:33:32.389 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-10-20 03:28:00.394405411 +0000 2023-10-20T03:33:32.389 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-10-20 03:28:00.394405411 +0000 2023-10-20T03:33:32.390 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-10-20T03:33:32.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-20T03:33:32.442 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-10-20T03:33:32.443 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-10-20T03:33:32.443 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000308834 s, 1.7 MB/s 2023-10-20T03:33:32.444 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-20T03:33:32.493 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2023-10-20T03:33:32.540 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-20T03:33:32.540 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:32.541 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 934 Links: 1 2023-10-20T03:33:32.541 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:32.541 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-10-20 03:28:01.374396657 +0000 2023-10-20T03:33:32.541 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-10-20 03:28:01.010399909 +0000 2023-10-20T03:33:32.541 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-10-20 03:28:01.010399909 +0000 2023-10-20T03:33:32.541 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-10-20T03:33:32.542 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-20T03:33:32.594 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-10-20T03:33:32.594 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-10-20T03:33:32.595 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000283306 s, 1.8 MB/s 2023-10-20T03:33:32.596 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-20T03:33:32.645 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-10-20T03:33:32.646 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:33:32.646 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2023-10-20T03:33:32.652 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-20T03:33:32.653 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2023-10-20T03:33:32.700 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-20T03:33:32.700 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:32.700 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 886 Links: 1 2023-10-20T03:33:32.700 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:32.701 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-20 03:27:57.358080946 +0000 2023-10-20T03:33:32.701 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-20 03:27:57.354080982 +0000 2023-10-20T03:33:32.701 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-20 03:27:57.354080982 +0000 2023-10-20T03:33:32.701 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2023-10-20T03:33:32.702 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-20T03:33:32.754 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-20T03:33:32.754 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-20T03:33:32.754 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000285308 s, 1.8 MB/s 2023-10-20T03:33:32.758 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-20T03:33:32.805 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2023-10-20T03:33:32.851 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-20T03:33:32.852 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:32.852 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 905 Links: 1 2023-10-20T03:33:32.852 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:32.852 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-20 03:27:58.294072585 +0000 2023-10-20T03:33:32.852 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-20 03:27:57.946075694 +0000 2023-10-20T03:33:32.853 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-20 03:27:57.946075694 +0000 2023-10-20T03:33:32.853 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2023-10-20T03:33:32.853 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-20T03:33:32.905 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-20T03:33:32.906 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-20T03:33:32.906 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000305704 s, 1.7 MB/s 2023-10-20T03:33:32.907 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-20T03:33:32.956 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2023-10-20T03:33:33.003 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-20T03:33:33.003 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:33.004 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 916 Links: 1 2023-10-20T03:33:33.004 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:33.004 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-20 03:27:58.882067332 +0000 2023-10-20T03:33:33.004 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-20 03:27:58.510070655 +0000 2023-10-20T03:33:33.004 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-20 03:27:58.510070655 +0000 2023-10-20T03:33:33.005 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2023-10-20T03:33:33.005 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-20T03:33:33.057 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-20T03:33:33.057 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-20T03:33:33.057 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.00030626 s, 1.7 MB/s 2023-10-20T03:33:33.058 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-20T03:33:33.108 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2023-10-20T03:33:33.155 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-20T03:33:33.156 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-20T03:33:33.156 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 933 Links: 1 2023-10-20T03:33:33.156 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-20T03:33:33.156 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2023-10-20 03:27:59.106065332 +0000 2023-10-20T03:33:33.156 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2023-10-20 03:27:59.102065367 +0000 2023-10-20T03:33:33.157 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2023-10-20 03:27:59.102065367 +0000 2023-10-20T03:33:33.157 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2023-10-20T03:33:33.157 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-20T03:33:33.209 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2023-10-20T03:33:33.209 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2023-10-20T03:33:33.209 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000236628 s, 2.2 MB/s 2023-10-20T03:33:33.210 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-20T03:33:33.260 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-10-20T03:33:33.261 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2023-10-20T03:33:33.261 INFO:tasks.ceph:Generating config... 2023-10-20T03:33:33.262 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-20T03:33:33.263 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-20T03:33:33.263 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2023-10-20T03:33:33.263 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-20T03:33:33.263 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-20T03:33:33.264 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-10-20T03:33:33.264 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-20T03:33:33.264 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-20T03:33:33.264 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-20T03:33:33.265 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-20T03:33:33.265 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-20T03:33:33.265 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-20T03:33:33.265 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-20T03:33:33.266 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-20T03:33:33.266 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-10-20T03:33:33.266 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2023-10-20T03:33:33.266 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-10-20T03:33:33.267 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-10-20T03:33:33.267 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-20T03:33:33.267 INFO:tasks.ceph:[osd] osd max backfills = 3 2023-10-20T03:33:33.267 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-10-20T03:33:33.268 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-10-20T03:33:33.268 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-10-20T03:33:33.268 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-20T03:33:33.269 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-10-20T03:33:33.269 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-10-20T03:33:33.269 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2023-10-20T03:33:33.269 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2023-10-20T03:33:33.270 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-20T03:33:33.270 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-20T03:33:33.270 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-10-20T03:33:33.270 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2023-10-20T03:33:33.271 INFO:tasks.ceph:[global] ms tcp read timeout = 5 2023-10-20T03:33:33.271 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2023-10-20T03:33:33.271 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2023-10-20T03:33:33.271 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-20T03:33:33.272 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-20T03:33:33.272 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-20T03:33:33.272 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-20T03:33:33.272 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-20T03:33:33.272 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-10-20T03:33:33.273 INFO:tasks.ceph:Setting up mon.a... 2023-10-20T03:33:33.273 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-20T03:33:33.309 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/ceph.keyring 2023-10-20T03:33:33.321 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-20T03:33:33.401 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-20T03:33:33.458 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.33'), ('mon.b', '172.21.15.40'), ('mon.c', '172.21.15.64')] 2023-10-20T03:33:33.459 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.33,172.21.15.40,172.21.15.64', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2023-10-20T03:33:33.460 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:33:33.461 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-20T03:33:33.506 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.33 --add b 172.21.15.40 --add c 172.21.15.64 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:33:33.593 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:33:33.593 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: generated fsid b920f165-e9e7-4c3e-8d7b-cfdc002ae589 2023-10-20T03:33:33.593 INFO:teuthology.orchestra.run.smithi033.stdout:setting min_mon_release = pacific 2023-10-20T03:33:33.593 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 0 2023-10-20T03:33:33.594 INFO:teuthology.orchestra.run.smithi033.stdout:fsid b920f165-e9e7-4c3e-8d7b-cfdc002ae589 2023-10-20T03:33:33.594 INFO:teuthology.orchestra.run.smithi033.stdout:last_changed 2023-10-20T03:33:33.594020+0000 2023-10-20T03:33:33.594 INFO:teuthology.orchestra.run.smithi033.stdout:created 2023-10-20T03:33:33.594020+0000 2023-10-20T03:33:33.594 INFO:teuthology.orchestra.run.smithi033.stdout:min_mon_release 16 (pacific) 2023-10-20T03:33:33.595 INFO:teuthology.orchestra.run.smithi033.stdout:election_strategy: 1 2023-10-20T03:33:33.595 INFO:teuthology.orchestra.run.smithi033.stdout:0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2023-10-20T03:33:33.595 INFO:teuthology.orchestra.run.smithi033.stdout:1: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.b 2023-10-20T03:33:33.595 INFO:teuthology.orchestra.run.smithi033.stdout:2: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.c 2023-10-20T03:33:33.595 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-20T03:33:33.596 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-20T03:33:33.641 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b920f165-e9e7-4c3e-8d7b-cfdc002ae589... 2023-10-20T03:33:33.642 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-20T03:33:33.687 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-20T03:33:33.689 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-20T03:33:33.692 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-20T03:33:33.716 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2023-10-20T03:33:33.716 INFO:teuthology.orchestra.run.smithi033.stdout: chdir = "" 2023-10-20T03:33:33.717 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-20T03:33:33.717 INFO:teuthology.orchestra.run.smithi033.stdout: auth supported = cephx 2023-10-20T03:33:33.717 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.717 INFO:teuthology.orchestra.run.smithi033.stdout: filestore xattr use omap = true 2023-10-20T03:33:33.717 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.718 INFO:teuthology.orchestra.run.smithi033.stdout: mon clock drift allowed = 1.000 2023-10-20T03:33:33.718 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.718 INFO:teuthology.orchestra.run.smithi033.stdout: osd crush chooseleaf type = 0 2023-10-20T03:33:33.718 INFO:teuthology.orchestra.run.smithi033.stdout: auth debug = true 2023-10-20T03:33:33.718 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.719 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on old message = true 2023-10-20T03:33:33.719 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on bug = true 2023-10-20T03:33:33.719 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.719 INFO:teuthology.orchestra.run.smithi033.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-20T03:33:33.719 INFO:teuthology.orchestra.run.smithi033.stdout: mon pg warn max object skew = 0 2023-10-20T03:33:33.720 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.720 INFO:teuthology.orchestra.run.smithi033.stdout: # disable pg_autoscaler by default for new pools 2023-10-20T03:33:33.720 INFO:teuthology.orchestra.run.smithi033.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-20T03:33:33.720 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.720 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default size = 2 2023-10-20T03:33:33.721 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.721 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow primary affinity = true 2023-10-20T03:33:33.721 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow pg remap = true 2023-10-20T03:33:33.721 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on legacy crush tunables = false 2023-10-20T03:33:33.721 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on crush straw calc version zero = false 2023-10-20T03:33:33.721 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on no sortbitwise = false 2023-10-20T03:33:33.722 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on osd down out interval zero = false 2023-10-20T03:33:33.722 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on too few osds = false 2023-10-20T03:33:33.722 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-20T03:33:33.722 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-20T03:33:33.722 INFO:teuthology.orchestra.run.smithi033.stdout: mon_allow_pool_size_one = true 2023-10-20T03:33:33.723 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.723 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-20T03:33:33.723 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.723 INFO:teuthology.orchestra.run.smithi033.stdout: osd default data pool replay window = 5 2023-10-20T03:33:33.723 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.724 INFO:teuthology.orchestra.run.smithi033.stdout: mon allow pool delete = true 2023-10-20T03:33:33.724 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.724 INFO:teuthology.orchestra.run.smithi033.stdout: mon cluster log file level = debug 2023-10-20T03:33:33.724 INFO:teuthology.orchestra.run.smithi033.stdout: debug asserts on shutdown = true 2023-10-20T03:33:33.724 INFO:teuthology.orchestra.run.smithi033.stdout: mon health detail to clog = false 2023-10-20T03:33:33.725 INFO:teuthology.orchestra.run.smithi033.stdout: mon host = "172.21.15.33,172.21.15.40,172.21.15.64" 2023-10-20T03:33:33.725 INFO:teuthology.orchestra.run.smithi033.stdout: mon client directed command retry = 5 2023-10-20T03:33:33.725 INFO:teuthology.orchestra.run.smithi033.stdout: mon election default strategy = 1 2023-10-20T03:33:33.725 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject socket failures = 2500 2023-10-20T03:33:33.725 INFO:teuthology.orchestra.run.smithi033.stdout: ms tcp read timeout = 5 2023-10-20T03:33:33.725 INFO:teuthology.orchestra.run.smithi033.stdout: osd_async_recovery_min_cost = 1 2023-10-20T03:33:33.726 INFO:teuthology.orchestra.run.smithi033.stdout: osd_object_clean_region_max_num_intervals = 1000 2023-10-20T03:33:33.726 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = b920f165-e9e7-4c3e-8d7b-cfdc002ae589 2023-10-20T03:33:33.726 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.726 INFO:teuthology.orchestra.run.smithi033.stdout:[osd] 2023-10-20T03:33:33.727 INFO:teuthology.orchestra.run.smithi033.stdout: osd journal size = 100 2023-10-20T03:33:33.727 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.727 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub load threshold = 5.0 2023-10-20T03:33:33.727 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub max interval = 120 2023-10-20T03:33:33.727 INFO:teuthology.orchestra.run.smithi033.stdout: osd mclock profile = high_recovery_ops 2023-10-20T03:33:33.727 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.728 INFO:teuthology.orchestra.run.smithi033.stdout: osd recover clone overlap = true 2023-10-20T03:33:33.728 INFO:teuthology.orchestra.run.smithi033.stdout: osd recovery max chunk = 1048576 2023-10-20T03:33:33.728 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.728 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug shutdown = true 2023-10-20T03:33:33.728 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug op order = true 2023-10-20T03:33:33.728 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify stray on activate = true 2023-10-20T03:33:33.729 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug trim objects = true 2023-10-20T03:33:33.729 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.729 INFO:teuthology.orchestra.run.smithi033.stdout: osd open classes on start = true 2023-10-20T03:33:33.729 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug pg log writeout = true 2023-10-20T03:33:33.729 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.730 INFO:teuthology.orchestra.run.smithi033.stdout: osd deep scrub update digest min age = 30 2023-10-20T03:33:33.730 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.730 INFO:teuthology.orchestra.run.smithi033.stdout: osd map max advance = 10 2023-10-20T03:33:33.730 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.730 INFO:teuthology.orchestra.run.smithi033.stdout: journal zero on create = true 2023-10-20T03:33:33.730 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.731 INFO:teuthology.orchestra.run.smithi033.stdout: filestore ondisk finisher threads = 3 2023-10-20T03:33:33.731 INFO:teuthology.orchestra.run.smithi033.stdout: filestore apply finisher threads = 3 2023-10-20T03:33:33.731 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.731 INFO:teuthology.orchestra.run.smithi033.stdout: bdev debug aio = true 2023-10-20T03:33:33.732 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug misdirected ops = true 2023-10-20T03:33:33.732 INFO:teuthology.orchestra.run.smithi033.stdout: debug monc = 20 2023-10-20T03:33:33.732 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore block size = 96636764160 2023-10-20T03:33:33.732 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore compression algorithm = zlib 2023-10-20T03:33:33.732 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore compression mode = aggressive 2023-10-20T03:33:33.733 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore fsck on mount = True 2023-10-20T03:33:33.733 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore zero block detection = True 2023-10-20T03:33:33.733 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluefs = 20 2023-10-20T03:33:33.733 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluestore = 20 2023-10-20T03:33:33.733 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-10-20T03:33:33.734 INFO:teuthology.orchestra.run.smithi033.stdout: debug osd = 20 2023-10-20T03:33:33.734 INFO:teuthology.orchestra.run.smithi033.stdout: debug rocksdb = 10 2023-10-20T03:33:33.734 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd backfillfull_ratio = 0.85 2023-10-20T03:33:33.734 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd full ratio = 0.9 2023-10-20T03:33:33.734 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd nearfull ratio = 0.8 2023-10-20T03:33:33.735 INFO:teuthology.orchestra.run.smithi033.stdout: osd blocked scrub grace period = 3600 2023-10-20T03:33:33.735 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug reject backfill probability = 0.3 2023-10-20T03:33:33.735 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify cached snaps = True 2023-10-20T03:33:33.735 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify missing on start = True 2023-10-20T03:33:33.735 INFO:teuthology.orchestra.run.smithi033.stdout: osd failsafe full ratio = 0.95 2023-10-20T03:33:33.736 INFO:teuthology.orchestra.run.smithi033.stdout: osd max backfills = 3 2023-10-20T03:33:33.736 INFO:teuthology.orchestra.run.smithi033.stdout: osd max markdown count = 1000 2023-10-20T03:33:33.736 INFO:teuthology.orchestra.run.smithi033.stdout: osd mclock override recovery settings = True 2023-10-20T03:33:33.736 INFO:teuthology.orchestra.run.smithi033.stdout: osd objectstore = bluestore 2023-10-20T03:33:33.736 INFO:teuthology.orchestra.run.smithi033.stdout: osd op queue = debug_random 2023-10-20T03:33:33.736 INFO:teuthology.orchestra.run.smithi033.stdout: osd op queue cut off = debug_random 2023-10-20T03:33:33.737 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub min interval = 60 2023-10-20T03:33:33.737 INFO:teuthology.orchestra.run.smithi033.stdout: osd shutdown pgref assert = True 2023-10-20T03:33:33.737 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.737 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr] 2023-10-20T03:33:33.737 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-10-20T03:33:33.738 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgr = 20 2023-10-20T03:33:33.738 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2023-10-20T03:33:33.738 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2023-10-20T03:33:33.738 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2023-10-20T03:33:33.738 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2023-10-20T03:33:33.739 INFO:teuthology.orchestra.run.smithi033.stdout: mgr/telemetry/nag = false 2023-10-20T03:33:33.739 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.739 INFO:teuthology.orchestra.run.smithi033.stdout:[mon] 2023-10-20T03:33:33.739 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-10-20T03:33:33.739 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2023-10-20T03:33:33.739 INFO:teuthology.orchestra.run.smithi033.stdout: debug paxos = 20 2023-10-20T03:33:33.740 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2023-10-20T03:33:33.740 INFO:teuthology.orchestra.run.smithi033.stdout: mon data avail warn = 5 2023-10-20T03:33:33.740 INFO:teuthology.orchestra.run.smithi033.stdout: mon mgr mkfs grace = 240 2023-10-20T03:33:33.740 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2023-10-20T03:33:33.740 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd reporter subtree level = osd 2023-10-20T03:33:33.741 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd prime pg temp = true 2023-10-20T03:33:33.741 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2023-10-20T03:33:33.741 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.741 INFO:teuthology.orchestra.run.smithi033.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-20T03:33:33.741 INFO:teuthology.orchestra.run.smithi033.stdout: auth mon ticket ttl = 660 # 11m 2023-10-20T03:33:33.741 INFO:teuthology.orchestra.run.smithi033.stdout: auth service ticket ttl = 240 # 4m 2023-10-20T03:33:33.742 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.742 INFO:teuthology.orchestra.run.smithi033.stdout: # don't complain about insecure global_id in the test suite 2023-10-20T03:33:33.742 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-20T03:33:33.742 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-20T03:33:33.742 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.743 INFO:teuthology.orchestra.run.smithi033.stdout: # 1m isn't quite enough 2023-10-20T03:33:33.743 INFO:teuthology.orchestra.run.smithi033.stdout: mon_down_mkfs_grace = 2m 2023-10-20T03:33:33.743 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.743 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_filestore_osds = false 2023-10-20T03:33:33.743 INFO:teuthology.orchestra.run.smithi033.stdout: mon scrub interval = 300 2023-10-20T03:33:33.743 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:33:33.744 INFO:teuthology.orchestra.run.smithi033.stdout:[client] 2023-10-20T03:33:33.744 INFO:teuthology.orchestra.run.smithi033.stdout: rgw cache enabled = true 2023-10-20T03:33:33.744 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable ops log = true 2023-10-20T03:33:33.744 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable usage log = true 2023-10-20T03:33:33.744 INFO:teuthology.orchestra.run.smithi033.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-20T03:33:33.745 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-20T03:33:33.745 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.a] 2023-10-20T03:33:33.745 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.b] 2023-10-20T03:33:33.745 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.c] 2023-10-20T03:33:33.748 INFO:teuthology.orchestra.run.smithi170.stdout:[global] 2023-10-20T03:33:33.748 INFO:teuthology.orchestra.run.smithi170.stdout: chdir = "" 2023-10-20T03:33:33.748 INFO:teuthology.orchestra.run.smithi170.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-20T03:33:33.748 INFO:teuthology.orchestra.run.smithi170.stdout: auth supported = cephx 2023-10-20T03:33:33.749 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.749 INFO:teuthology.orchestra.run.smithi170.stdout: filestore xattr use omap = true 2023-10-20T03:33:33.749 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.749 INFO:teuthology.orchestra.run.smithi170.stdout: mon clock drift allowed = 1.000 2023-10-20T03:33:33.749 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.749 INFO:teuthology.orchestra.run.smithi170.stdout: osd crush chooseleaf type = 0 2023-10-20T03:33:33.750 INFO:teuthology.orchestra.run.smithi170.stdout: auth debug = true 2023-10-20T03:33:33.750 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.750 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on old message = true 2023-10-20T03:33:33.750 INFO:teuthology.orchestra.run.smithi170.stdout: ms die on bug = true 2023-10-20T03:33:33.750 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.751 INFO:teuthology.orchestra.run.smithi170.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-20T03:33:33.751 INFO:teuthology.orchestra.run.smithi170.stdout: mon pg warn max object skew = 0 2023-10-20T03:33:33.751 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.751 INFO:teuthology.orchestra.run.smithi170.stdout: # disable pg_autoscaler by default for new pools 2023-10-20T03:33:33.751 INFO:teuthology.orchestra.run.smithi170.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-20T03:33:33.752 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.752 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default size = 2 2023-10-20T03:33:33.752 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.752 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow primary affinity = true 2023-10-20T03:33:33.752 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd allow pg remap = true 2023-10-20T03:33:33.753 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on legacy crush tunables = false 2023-10-20T03:33:33.753 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on crush straw calc version zero = false 2023-10-20T03:33:33.753 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on no sortbitwise = false 2023-10-20T03:33:33.753 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on osd down out interval zero = false 2023-10-20T03:33:33.753 INFO:teuthology.orchestra.run.smithi170.stdout: mon warn on too few osds = false 2023-10-20T03:33:33.754 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-20T03:33:33.754 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-20T03:33:33.754 INFO:teuthology.orchestra.run.smithi170.stdout: mon_allow_pool_size_one = true 2023-10-20T03:33:33.754 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.754 INFO:teuthology.orchestra.run.smithi170.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-20T03:33:33.755 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.755 INFO:teuthology.orchestra.run.smithi170.stdout: osd default data pool replay window = 5 2023-10-20T03:33:33.755 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.755 INFO:teuthology.orchestra.run.smithi170.stdout: mon allow pool delete = true 2023-10-20T03:33:33.755 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.755 INFO:teuthology.orchestra.run.smithi170.stdout: mon cluster log file level = debug 2023-10-20T03:33:33.756 INFO:teuthology.orchestra.run.smithi170.stdout: debug asserts on shutdown = true 2023-10-20T03:33:33.756 INFO:teuthology.orchestra.run.smithi170.stdout: mon health detail to clog = false 2023-10-20T03:33:33.756 INFO:teuthology.orchestra.run.smithi170.stdout: mon host = "172.21.15.33,172.21.15.40,172.21.15.64" 2023-10-20T03:33:33.756 INFO:teuthology.orchestra.run.smithi170.stdout: mon client directed command retry = 5 2023-10-20T03:33:33.756 INFO:teuthology.orchestra.run.smithi170.stdout: mon election default strategy = 1 2023-10-20T03:33:33.757 INFO:teuthology.orchestra.run.smithi170.stdout: ms inject socket failures = 2500 2023-10-20T03:33:33.757 INFO:teuthology.orchestra.run.smithi170.stdout: ms tcp read timeout = 5 2023-10-20T03:33:33.757 INFO:teuthology.orchestra.run.smithi170.stdout: osd_async_recovery_min_cost = 1 2023-10-20T03:33:33.757 INFO:teuthology.orchestra.run.smithi170.stdout: osd_object_clean_region_max_num_intervals = 1000 2023-10-20T03:33:33.757 INFO:teuthology.orchestra.run.smithi170.stdout: fsid = b920f165-e9e7-4c3e-8d7b-cfdc002ae589 2023-10-20T03:33:33.757 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.758 INFO:teuthology.orchestra.run.smithi170.stdout:[osd] 2023-10-20T03:33:33.758 INFO:teuthology.orchestra.run.smithi170.stdout: osd journal size = 100 2023-10-20T03:33:33.758 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.758 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub load threshold = 5.0 2023-10-20T03:33:33.758 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub max interval = 120 2023-10-20T03:33:33.759 INFO:teuthology.orchestra.run.smithi170.stdout: osd mclock profile = high_recovery_ops 2023-10-20T03:33:33.759 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.759 INFO:teuthology.orchestra.run.smithi170.stdout: osd recover clone overlap = true 2023-10-20T03:33:33.759 INFO:teuthology.orchestra.run.smithi170.stdout: osd recovery max chunk = 1048576 2023-10-20T03:33:33.759 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.759 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug shutdown = true 2023-10-20T03:33:33.760 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug op order = true 2023-10-20T03:33:33.760 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify stray on activate = true 2023-10-20T03:33:33.760 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug trim objects = true 2023-10-20T03:33:33.760 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.760 INFO:teuthology.orchestra.run.smithi170.stdout: osd open classes on start = true 2023-10-20T03:33:33.761 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug pg log writeout = true 2023-10-20T03:33:33.761 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.761 INFO:teuthology.orchestra.run.smithi170.stdout: osd deep scrub update digest min age = 30 2023-10-20T03:33:33.761 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.761 INFO:teuthology.orchestra.run.smithi170.stdout: osd map max advance = 10 2023-10-20T03:33:33.762 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.762 INFO:teuthology.orchestra.run.smithi170.stdout: journal zero on create = true 2023-10-20T03:33:33.762 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.762 INFO:teuthology.orchestra.run.smithi170.stdout: filestore ondisk finisher threads = 3 2023-10-20T03:33:33.762 INFO:teuthology.orchestra.run.smithi170.stdout: filestore apply finisher threads = 3 2023-10-20T03:33:33.763 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.763 INFO:teuthology.orchestra.run.smithi170.stdout: bdev debug aio = true 2023-10-20T03:33:33.763 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug misdirected ops = true 2023-10-20T03:33:33.763 INFO:teuthology.orchestra.run.smithi170.stdout: debug monc = 20 2023-10-20T03:33:33.763 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore block size = 96636764160 2023-10-20T03:33:33.764 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore compression algorithm = zlib 2023-10-20T03:33:33.764 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore compression mode = aggressive 2023-10-20T03:33:33.764 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore fsck on mount = True 2023-10-20T03:33:33.764 INFO:teuthology.orchestra.run.smithi170.stdout: bluestore zero block detection = True 2023-10-20T03:33:33.764 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluefs = 20 2023-10-20T03:33:33.764 INFO:teuthology.orchestra.run.smithi170.stdout: debug bluestore = 20 2023-10-20T03:33:33.765 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-20T03:33:33.765 INFO:teuthology.orchestra.run.smithi170.stdout: debug osd = 20 2023-10-20T03:33:33.765 INFO:teuthology.orchestra.run.smithi170.stdout: debug rocksdb = 10 2023-10-20T03:33:33.765 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd backfillfull_ratio = 0.85 2023-10-20T03:33:33.765 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd full ratio = 0.9 2023-10-20T03:33:33.766 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd nearfull ratio = 0.8 2023-10-20T03:33:33.766 INFO:teuthology.orchestra.run.smithi170.stdout: osd blocked scrub grace period = 3600 2023-10-20T03:33:33.766 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug reject backfill probability = 0.3 2023-10-20T03:33:33.766 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify cached snaps = True 2023-10-20T03:33:33.766 INFO:teuthology.orchestra.run.smithi170.stdout: osd debug verify missing on start = True 2023-10-20T03:33:33.767 INFO:teuthology.orchestra.run.smithi170.stdout: osd failsafe full ratio = 0.95 2023-10-20T03:33:33.767 INFO:teuthology.orchestra.run.smithi170.stdout: osd max backfills = 3 2023-10-20T03:33:33.767 INFO:teuthology.orchestra.run.smithi170.stdout: osd max markdown count = 1000 2023-10-20T03:33:33.767 INFO:teuthology.orchestra.run.smithi170.stdout: osd mclock override recovery settings = True 2023-10-20T03:33:33.767 INFO:teuthology.orchestra.run.smithi170.stdout: osd objectstore = bluestore 2023-10-20T03:33:33.768 INFO:teuthology.orchestra.run.smithi170.stdout: osd op queue = debug_random 2023-10-20T03:33:33.768 INFO:teuthology.orchestra.run.smithi170.stdout: osd op queue cut off = debug_random 2023-10-20T03:33:33.768 INFO:teuthology.orchestra.run.smithi170.stdout: osd scrub min interval = 60 2023-10-20T03:33:33.768 INFO:teuthology.orchestra.run.smithi170.stdout: osd shutdown pgref assert = True 2023-10-20T03:33:33.768 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.768 INFO:teuthology.orchestra.run.smithi170.stdout:[mgr] 2023-10-20T03:33:33.769 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-20T03:33:33.769 INFO:teuthology.orchestra.run.smithi170.stdout: debug mgr = 20 2023-10-20T03:33:33.769 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-10-20T03:33:33.769 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-10-20T03:33:33.769 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-10-20T03:33:33.769 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-10-20T03:33:33.770 INFO:teuthology.orchestra.run.smithi170.stdout: mgr/telemetry/nag = false 2023-10-20T03:33:33.770 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.770 INFO:teuthology.orchestra.run.smithi170.stdout:[mon] 2023-10-20T03:33:33.770 INFO:teuthology.orchestra.run.smithi170.stdout: debug ms = 1 2023-10-20T03:33:33.770 INFO:teuthology.orchestra.run.smithi170.stdout: debug mon = 20 2023-10-20T03:33:33.771 INFO:teuthology.orchestra.run.smithi170.stdout: debug paxos = 20 2023-10-20T03:33:33.771 INFO:teuthology.orchestra.run.smithi170.stdout: debug auth = 20 2023-10-20T03:33:33.771 INFO:teuthology.orchestra.run.smithi170.stdout: mon data avail warn = 5 2023-10-20T03:33:33.771 INFO:teuthology.orchestra.run.smithi170.stdout: mon mgr mkfs grace = 240 2023-10-20T03:33:33.771 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min pgs per osd = 4 2023-10-20T03:33:33.772 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd reporter subtree level = osd 2023-10-20T03:33:33.772 INFO:teuthology.orchestra.run.smithi170.stdout: mon osd prime pg temp = true 2023-10-20T03:33:33.772 INFO:teuthology.orchestra.run.smithi170.stdout: mon reweight min bytes per osd = 10 2023-10-20T03:33:33.772 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.772 INFO:teuthology.orchestra.run.smithi170.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-20T03:33:33.773 INFO:teuthology.orchestra.run.smithi170.stdout: auth mon ticket ttl = 660 # 11m 2023-10-20T03:33:33.773 INFO:teuthology.orchestra.run.smithi170.stdout: auth service ticket ttl = 240 # 4m 2023-10-20T03:33:33.773 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.773 INFO:teuthology.orchestra.run.smithi170.stdout: # don't complain about insecure global_id in the test suite 2023-10-20T03:33:33.773 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-20T03:33:33.774 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-20T03:33:33.774 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.774 INFO:teuthology.orchestra.run.smithi170.stdout: # 1m isn't quite enough 2023-10-20T03:33:33.774 INFO:teuthology.orchestra.run.smithi170.stdout: mon_down_mkfs_grace = 2m 2023-10-20T03:33:33.774 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.775 INFO:teuthology.orchestra.run.smithi170.stdout: mon_warn_on_filestore_osds = false 2023-10-20T03:33:33.775 INFO:teuthology.orchestra.run.smithi170.stdout: mon scrub interval = 300 2023-10-20T03:33:33.775 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-10-20T03:33:33.775 INFO:teuthology.orchestra.run.smithi170.stdout:[client] 2023-10-20T03:33:33.775 INFO:teuthology.orchestra.run.smithi170.stdout: rgw cache enabled = true 2023-10-20T03:33:33.776 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable ops log = true 2023-10-20T03:33:33.776 INFO:teuthology.orchestra.run.smithi170.stdout: rgw enable usage log = true 2023-10-20T03:33:33.776 INFO:teuthology.orchestra.run.smithi170.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-20T03:33:33.776 INFO:teuthology.orchestra.run.smithi170.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-20T03:33:33.776 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.a] 2023-10-20T03:33:33.776 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.b] 2023-10-20T03:33:33.777 INFO:teuthology.orchestra.run.smithi170.stdout:[mon.c] 2023-10-20T03:33:33.777 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2023-10-20T03:33:33.777 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2023-10-20T03:33:33.777 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-20T03:33:33.778 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2023-10-20T03:33:33.778 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.778 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2023-10-20T03:33:33.781 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.781 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2023-10-20T03:33:33.781 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.782 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2023-10-20T03:33:33.782 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2023-10-20T03:33:33.782 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.782 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2023-10-20T03:33:33.782 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2023-10-20T03:33:33.783 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.783 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-20T03:33:33.783 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2023-10-20T03:33:33.783 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.783 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2023-10-20T03:33:33.784 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-20T03:33:33.784 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.784 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2023-10-20T03:33:33.784 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.784 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2023-10-20T03:33:33.785 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2023-10-20T03:33:33.785 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2023-10-20T03:33:33.785 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2023-10-20T03:33:33.785 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2023-10-20T03:33:33.785 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2023-10-20T03:33:33.785 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2023-10-20T03:33:33.786 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-20T03:33:33.786 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-20T03:33:33.786 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2023-10-20T03:33:33.786 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.786 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-20T03:33:33.787 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.787 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2023-10-20T03:33:33.787 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.787 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2023-10-20T03:33:33.787 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.787 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2023-10-20T03:33:33.788 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2023-10-20T03:33:33.788 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2023-10-20T03:33:33.788 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = "172.21.15.33,172.21.15.40,172.21.15.64" 2023-10-20T03:33:33.788 INFO:teuthology.orchestra.run.smithi064.stdout: mon client directed command retry = 5 2023-10-20T03:33:33.788 INFO:teuthology.orchestra.run.smithi064.stdout: mon election default strategy = 1 2023-10-20T03:33:33.788 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject socket failures = 2500 2023-10-20T03:33:33.789 INFO:teuthology.orchestra.run.smithi064.stdout: ms tcp read timeout = 5 2023-10-20T03:33:33.789 INFO:teuthology.orchestra.run.smithi064.stdout: osd_async_recovery_min_cost = 1 2023-10-20T03:33:33.789 INFO:teuthology.orchestra.run.smithi064.stdout: osd_object_clean_region_max_num_intervals = 1000 2023-10-20T03:33:33.789 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = b920f165-e9e7-4c3e-8d7b-cfdc002ae589 2023-10-20T03:33:33.789 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.789 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2023-10-20T03:33:33.790 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2023-10-20T03:33:33.790 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.790 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2023-10-20T03:33:33.790 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 120 2023-10-20T03:33:33.790 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2023-10-20T03:33:33.791 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2023-10-20T03:33:33.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2023-10-20T03:33:33.791 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2023-10-20T03:33:33.791 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2023-10-20T03:33:33.792 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2023-10-20T03:33:33.792 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug trim objects = true 2023-10-20T03:33:33.792 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.792 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2023-10-20T03:33:33.792 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2023-10-20T03:33:33.792 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.793 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2023-10-20T03:33:33.793 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.793 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2023-10-20T03:33:33.793 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.793 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2023-10-20T03:33:33.794 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.794 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2023-10-20T03:33:33.794 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2023-10-20T03:33:33.794 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.794 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2023-10-20T03:33:33.794 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2023-10-20T03:33:33.795 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2023-10-20T03:33:33.795 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2023-10-20T03:33:33.795 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression algorithm = zlib 2023-10-20T03:33:33.795 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression mode = aggressive 2023-10-20T03:33:33.795 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2023-10-20T03:33:33.796 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore zero block detection = True 2023-10-20T03:33:33.796 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 20 2023-10-20T03:33:33.796 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 20 2023-10-20T03:33:33.796 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-10-20T03:33:33.796 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2023-10-20T03:33:33.796 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 10 2023-10-20T03:33:33.797 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2023-10-20T03:33:33.797 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2023-10-20T03:33:33.797 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2023-10-20T03:33:33.797 INFO:teuthology.orchestra.run.smithi064.stdout: osd blocked scrub grace period = 3600 2023-10-20T03:33:33.797 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug reject backfill probability = 0.3 2023-10-20T03:33:33.798 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify cached snaps = True 2023-10-20T03:33:33.798 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify missing on start = True 2023-10-20T03:33:33.798 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2023-10-20T03:33:33.798 INFO:teuthology.orchestra.run.smithi064.stdout: osd max backfills = 3 2023-10-20T03:33:33.798 INFO:teuthology.orchestra.run.smithi064.stdout: osd max markdown count = 1000 2023-10-20T03:33:33.798 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock override recovery settings = True 2023-10-20T03:33:33.799 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2023-10-20T03:33:33.799 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue = debug_random 2023-10-20T03:33:33.799 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue cut off = debug_random 2023-10-20T03:33:33.799 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub min interval = 60 2023-10-20T03:33:33.799 INFO:teuthology.orchestra.run.smithi064.stdout: osd shutdown pgref assert = True 2023-10-20T03:33:33.800 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.800 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2023-10-20T03:33:33.800 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-10-20T03:33:33.800 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2023-10-20T03:33:33.800 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2023-10-20T03:33:33.800 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2023-10-20T03:33:33.801 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2023-10-20T03:33:33.801 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2023-10-20T03:33:33.801 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2023-10-20T03:33:33.801 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.801 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2023-10-20T03:33:33.802 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-10-20T03:33:33.802 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2023-10-20T03:33:33.802 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2023-10-20T03:33:33.802 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2023-10-20T03:33:33.802 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2023-10-20T03:33:33.803 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2023-10-20T03:33:33.803 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2023-10-20T03:33:33.803 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2023-10-20T03:33:33.803 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2023-10-20T03:33:33.803 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2023-10-20T03:33:33.804 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.804 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-20T03:33:33.804 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2023-10-20T03:33:33.804 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2023-10-20T03:33:33.804 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.804 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2023-10-20T03:33:33.805 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-20T03:33:33.805 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-20T03:33:33.805 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.805 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2023-10-20T03:33:33.805 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2023-10-20T03:33:33.805 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.806 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2023-10-20T03:33:33.806 INFO:teuthology.orchestra.run.smithi064.stdout: mon scrub interval = 300 2023-10-20T03:33:33.806 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-10-20T03:33:33.806 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2023-10-20T03:33:33.806 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2023-10-20T03:33:33.806 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2023-10-20T03:33:33.807 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2023-10-20T03:33:33.807 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-20T03:33:33.807 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-20T03:33:33.807 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2023-10-20T03:33:33.807 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.b] 2023-10-20T03:33:33.807 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.c] 2023-10-20T03:33:33.808 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2023-10-20T03:33:33.808 INFO:teuthology.orchestra.run.smithi040.stdout: chdir = "" 2023-10-20T03:33:33.808 INFO:teuthology.orchestra.run.smithi040.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-20T03:33:33.808 INFO:teuthology.orchestra.run.smithi040.stdout: auth supported = cephx 2023-10-20T03:33:33.808 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.808 INFO:teuthology.orchestra.run.smithi040.stdout: filestore xattr use omap = true 2023-10-20T03:33:33.809 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.809 INFO:teuthology.orchestra.run.smithi040.stdout: mon clock drift allowed = 1.000 2023-10-20T03:33:33.809 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.809 INFO:teuthology.orchestra.run.smithi040.stdout: osd crush chooseleaf type = 0 2023-10-20T03:33:33.809 INFO:teuthology.orchestra.run.smithi040.stdout: auth debug = true 2023-10-20T03:33:33.809 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.810 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on old message = true 2023-10-20T03:33:33.810 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on bug = true 2023-10-20T03:33:33.810 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.810 INFO:teuthology.orchestra.run.smithi040.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-20T03:33:33.810 INFO:teuthology.orchestra.run.smithi040.stdout: mon pg warn max object skew = 0 2023-10-20T03:33:33.811 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.811 INFO:teuthology.orchestra.run.smithi040.stdout: # disable pg_autoscaler by default for new pools 2023-10-20T03:33:33.811 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-20T03:33:33.811 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.811 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default size = 2 2023-10-20T03:33:33.811 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.812 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow primary affinity = true 2023-10-20T03:33:33.812 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow pg remap = true 2023-10-20T03:33:33.812 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on legacy crush tunables = false 2023-10-20T03:33:33.812 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on crush straw calc version zero = false 2023-10-20T03:33:33.812 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on no sortbitwise = false 2023-10-20T03:33:33.812 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on osd down out interval zero = false 2023-10-20T03:33:33.813 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on too few osds = false 2023-10-20T03:33:33.813 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-20T03:33:33.813 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-20T03:33:33.813 INFO:teuthology.orchestra.run.smithi040.stdout: mon_allow_pool_size_one = true 2023-10-20T03:33:33.813 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.814 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-20T03:33:33.814 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.814 INFO:teuthology.orchestra.run.smithi040.stdout: osd default data pool replay window = 5 2023-10-20T03:33:33.814 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.814 INFO:teuthology.orchestra.run.smithi040.stdout: mon allow pool delete = true 2023-10-20T03:33:33.815 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.815 INFO:teuthology.orchestra.run.smithi040.stdout: mon cluster log file level = debug 2023-10-20T03:33:33.815 INFO:teuthology.orchestra.run.smithi040.stdout: debug asserts on shutdown = true 2023-10-20T03:33:33.815 INFO:teuthology.orchestra.run.smithi040.stdout: mon health detail to clog = false 2023-10-20T03:33:33.815 INFO:teuthology.orchestra.run.smithi040.stdout: mon host = "172.21.15.33,172.21.15.40,172.21.15.64" 2023-10-20T03:33:33.815 INFO:teuthology.orchestra.run.smithi040.stdout: mon client directed command retry = 5 2023-10-20T03:33:33.816 INFO:teuthology.orchestra.run.smithi040.stdout: mon election default strategy = 1 2023-10-20T03:33:33.816 INFO:teuthology.orchestra.run.smithi040.stdout: ms inject socket failures = 2500 2023-10-20T03:33:33.816 INFO:teuthology.orchestra.run.smithi040.stdout: ms tcp read timeout = 5 2023-10-20T03:33:33.816 INFO:teuthology.orchestra.run.smithi040.stdout: osd_async_recovery_min_cost = 1 2023-10-20T03:33:33.816 INFO:teuthology.orchestra.run.smithi040.stdout: osd_object_clean_region_max_num_intervals = 1000 2023-10-20T03:33:33.816 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = b920f165-e9e7-4c3e-8d7b-cfdc002ae589 2023-10-20T03:33:33.817 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.817 INFO:teuthology.orchestra.run.smithi040.stdout:[osd] 2023-10-20T03:33:33.817 INFO:teuthology.orchestra.run.smithi040.stdout: osd journal size = 100 2023-10-20T03:33:33.817 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.817 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub load threshold = 5.0 2023-10-20T03:33:33.817 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub max interval = 120 2023-10-20T03:33:33.818 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock profile = high_recovery_ops 2023-10-20T03:33:33.818 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.818 INFO:teuthology.orchestra.run.smithi040.stdout: osd recover clone overlap = true 2023-10-20T03:33:33.818 INFO:teuthology.orchestra.run.smithi040.stdout: osd recovery max chunk = 1048576 2023-10-20T03:33:33.818 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.819 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug shutdown = true 2023-10-20T03:33:33.819 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug op order = true 2023-10-20T03:33:33.819 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify stray on activate = true 2023-10-20T03:33:33.819 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug trim objects = true 2023-10-20T03:33:33.819 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.820 INFO:teuthology.orchestra.run.smithi040.stdout: osd open classes on start = true 2023-10-20T03:33:33.820 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug pg log writeout = true 2023-10-20T03:33:33.820 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.820 INFO:teuthology.orchestra.run.smithi040.stdout: osd deep scrub update digest min age = 30 2023-10-20T03:33:33.820 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.820 INFO:teuthology.orchestra.run.smithi040.stdout: osd map max advance = 10 2023-10-20T03:33:33.821 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.821 INFO:teuthology.orchestra.run.smithi040.stdout: journal zero on create = true 2023-10-20T03:33:33.821 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.821 INFO:teuthology.orchestra.run.smithi040.stdout: filestore ondisk finisher threads = 3 2023-10-20T03:33:33.821 INFO:teuthology.orchestra.run.smithi040.stdout: filestore apply finisher threads = 3 2023-10-20T03:33:33.821 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.822 INFO:teuthology.orchestra.run.smithi040.stdout: bdev debug aio = true 2023-10-20T03:33:33.822 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug misdirected ops = true 2023-10-20T03:33:33.822 INFO:teuthology.orchestra.run.smithi040.stdout: debug monc = 20 2023-10-20T03:33:33.822 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore block size = 96636764160 2023-10-20T03:33:33.822 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore compression algorithm = zlib 2023-10-20T03:33:33.823 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore compression mode = aggressive 2023-10-20T03:33:33.823 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore fsck on mount = True 2023-10-20T03:33:33.823 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore zero block detection = True 2023-10-20T03:33:33.823 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluefs = 20 2023-10-20T03:33:33.823 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluestore = 20 2023-10-20T03:33:33.824 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2023-10-20T03:33:33.824 INFO:teuthology.orchestra.run.smithi040.stdout: debug osd = 20 2023-10-20T03:33:33.824 INFO:teuthology.orchestra.run.smithi040.stdout: debug rocksdb = 10 2023-10-20T03:33:33.824 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd backfillfull_ratio = 0.85 2023-10-20T03:33:33.824 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd full ratio = 0.9 2023-10-20T03:33:33.825 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd nearfull ratio = 0.8 2023-10-20T03:33:33.825 INFO:teuthology.orchestra.run.smithi040.stdout: osd blocked scrub grace period = 3600 2023-10-20T03:33:33.825 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug reject backfill probability = 0.3 2023-10-20T03:33:33.825 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify cached snaps = True 2023-10-20T03:33:33.825 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify missing on start = True 2023-10-20T03:33:33.825 INFO:teuthology.orchestra.run.smithi040.stdout: osd failsafe full ratio = 0.95 2023-10-20T03:33:33.826 INFO:teuthology.orchestra.run.smithi040.stdout: osd max backfills = 3 2023-10-20T03:33:33.826 INFO:teuthology.orchestra.run.smithi040.stdout: osd max markdown count = 1000 2023-10-20T03:33:33.826 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock override recovery settings = True 2023-10-20T03:33:33.826 INFO:teuthology.orchestra.run.smithi040.stdout: osd objectstore = bluestore 2023-10-20T03:33:33.826 INFO:teuthology.orchestra.run.smithi040.stdout: osd op queue = debug_random 2023-10-20T03:33:33.827 INFO:teuthology.orchestra.run.smithi040.stdout: osd op queue cut off = debug_random 2023-10-20T03:33:33.827 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub min interval = 60 2023-10-20T03:33:33.827 INFO:teuthology.orchestra.run.smithi040.stdout: osd shutdown pgref assert = True 2023-10-20T03:33:33.827 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.827 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr] 2023-10-20T03:33:33.828 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2023-10-20T03:33:33.828 INFO:teuthology.orchestra.run.smithi040.stdout: debug mgr = 20 2023-10-20T03:33:33.828 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2023-10-20T03:33:33.828 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2023-10-20T03:33:33.828 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2023-10-20T03:33:33.828 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2023-10-20T03:33:33.829 INFO:teuthology.orchestra.run.smithi040.stdout: mgr/telemetry/nag = false 2023-10-20T03:33:33.829 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.829 INFO:teuthology.orchestra.run.smithi040.stdout:[mon] 2023-10-20T03:33:33.829 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2023-10-20T03:33:33.829 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2023-10-20T03:33:33.830 INFO:teuthology.orchestra.run.smithi040.stdout: debug paxos = 20 2023-10-20T03:33:33.830 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2023-10-20T03:33:33.830 INFO:teuthology.orchestra.run.smithi040.stdout: mon data avail warn = 5 2023-10-20T03:33:33.830 INFO:teuthology.orchestra.run.smithi040.stdout: mon mgr mkfs grace = 240 2023-10-20T03:33:33.830 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2023-10-20T03:33:33.830 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd reporter subtree level = osd 2023-10-20T03:33:33.831 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd prime pg temp = true 2023-10-20T03:33:33.831 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2023-10-20T03:33:33.831 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.831 INFO:teuthology.orchestra.run.smithi040.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-20T03:33:33.831 INFO:teuthology.orchestra.run.smithi040.stdout: auth mon ticket ttl = 660 # 11m 2023-10-20T03:33:33.831 INFO:teuthology.orchestra.run.smithi040.stdout: auth service ticket ttl = 240 # 4m 2023-10-20T03:33:33.832 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.832 INFO:teuthology.orchestra.run.smithi040.stdout: # don't complain about insecure global_id in the test suite 2023-10-20T03:33:33.832 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-20T03:33:33.832 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-20T03:33:33.832 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.833 INFO:teuthology.orchestra.run.smithi040.stdout: # 1m isn't quite enough 2023-10-20T03:33:33.833 INFO:teuthology.orchestra.run.smithi040.stdout: mon_down_mkfs_grace = 2m 2023-10-20T03:33:33.833 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.833 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_filestore_osds = false 2023-10-20T03:33:33.833 INFO:teuthology.orchestra.run.smithi040.stdout: mon scrub interval = 300 2023-10-20T03:33:33.833 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-20T03:33:33.834 INFO:teuthology.orchestra.run.smithi040.stdout:[client] 2023-10-20T03:33:33.834 INFO:teuthology.orchestra.run.smithi040.stdout: rgw cache enabled = true 2023-10-20T03:33:33.834 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable ops log = true 2023-10-20T03:33:33.834 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable usage log = true 2023-10-20T03:33:33.835 INFO:teuthology.orchestra.run.smithi040.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-20T03:33:33.835 INFO:teuthology.orchestra.run.smithi040.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-20T03:33:33.835 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.a] 2023-10-20T03:33:33.835 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.b] 2023-10-20T03:33:33.835 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.c] 2023-10-20T03:33:33.836 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-20T03:33:33.836 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-20T03:33:33.888 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-20T03:33:33.888 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:33:33.888 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-20T03:33:33.894 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:33:33.895 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-20T03:33:33.941 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi033.front.sepia.ceph.com 2023-10-20T03:33:33.942 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:33:33.942 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-20T03:33:33.942 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-20T03:33:34.006 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:33:34.006 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:33:34.054 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi040.front.sepia.ceph.com 2023-10-20T03:33:34.055 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:33:34.055 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-20T03:33:34.055 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-20T03:33:34.080 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:33:34.080 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:33:34.127 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2023-10-20T03:33:34.128 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:33:34.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-20T03:33:34.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-20T03:33:34.153 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:33:34.153 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:33:34.200 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi170.front.sepia.ceph.com 2023-10-20T03:33:34.200 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:33:34.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-20T03:33:34.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-20T03:33:34.223 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:33:34.224 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:33:34.271 INFO:tasks.ceph:Setting up mon nodes... 2023-10-20T03:33:34.271 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-20T03:33:34.272 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-20T03:33:34.317 INFO:teuthology.orchestra.run.smithi033.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-20T03:33:34.320 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-20T03:33:34.361 INFO:teuthology.orchestra.run.smithi170.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-20T03:33:34.375 INFO:tasks.ceph:Setting up mds nodes... 2023-10-20T03:33:34.376 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-20T03:33:34.376 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-20T03:33:34.451 INFO:teuthology.orchestra.run.smithi170.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-20T03:33:34.461 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-20T03:33:34.461 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2023-10-20T03:33:34.462 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-20T03:33:34.477 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-10-20T03:33:34.477 INFO:tasks.ceph:role: osd.0 2023-10-20T03:33:34.477 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-20T03:33:34.478 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-20T03:33:34.541 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:33:34.542 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:33:34.542 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:33:34.542 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:33:34.542 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:33:34.543 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:33:34.543 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:33:34.543 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:33:34.543 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:33:34.543 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:33:34.999 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-10-20T03:33:35.001 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-10-20T03:33:35.002 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-20T03:33:35.141 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-20T03:33:35.153 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:33:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:33:35.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-20T03:33:35.209 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-10-20T03:33:35.209 INFO:tasks.ceph:role: osd.4 2023-10-20T03:33:35.210 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-20T03:33:35.210 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-20T03:33:35.272 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:33:35.273 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:33:35.273 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:33:35.273 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:33:35.273 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:33:35.273 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:33:35.274 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:33:35.274 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:33:35.274 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:33:35.274 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:33:35.697 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-10-20T03:33:35.699 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-10-20T03:33:35.700 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2023-10-20T03:33:35.760 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-20T03:33:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:33:35.811 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:33:35.811 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2023-10-20T03:33:35.863 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-10-20T03:33:35.864 INFO:tasks.ceph:role: osd.8 2023-10-20T03:33:35.864 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-20T03:33:35.864 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-20T03:33:35.929 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:33:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:33:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:33:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:33:35.931 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:33:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:33:35.931 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:33:35.932 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:33:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:33:35.932 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:33:36.390 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-10-20T03:33:36.392 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-10-20T03:33:36.392 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2023-10-20T03:33:36.486 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2023-10-20T03:33:36.496 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:33:36.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:33:36.497 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2023-10-20T03:33:36.551 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-10-20T03:33:36.552 INFO:tasks.ceph:role: osd.12 2023-10-20T03:33:36.552 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-20T03:33:36.552 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-20T03:33:36.614 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:33:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:33:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:33:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:33:36.615 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:33:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:33:36.615 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:33:36.616 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:33:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:33:36.616 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:33:37.077 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-10-20T03:33:37.079 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-10-20T03:33:37.079 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2023-10-20T03:33:37.224 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2023-10-20T03:33:37.236 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:33:37.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:33:37.237 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:33:37.348 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:37.343+0000 7fc3ed715640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-20T03:33:37.350 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:37.347+0000 7fc3ed715640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-20T03:33:37.351 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:37.347+0000 7fc3ed715640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-20T03:33:37.351 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:37.347+0000 7fc3ed715640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-20T03:33:37.351 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:37.347+0000 7fc3ed715640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-20T03:33:37.357 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:37.355+0000 7fc3ed715640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-20T03:33:40.378 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-20T03:33:40.391 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:33:40.475 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:40.471+0000 7fdf043e0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-20T03:33:40.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:40.471+0000 7fdf043e0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-20T03:33:40.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:40.471+0000 7fdf043e0640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-20T03:33:40.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:40.471+0000 7fdf043e0640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-20T03:33:40.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:40.471+0000 7fdf043e0640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-20T03:33:40.477 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:40.471+0000 7fdf043e0640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-20T03:33:43.473 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-20T03:33:43.531 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:33:43.616 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:43.611+0000 7fbd9d93e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2023-10-20T03:33:43.617 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:43.611+0000 7fbd9d93e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2023-10-20T03:33:43.617 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:43.611+0000 7fbd9d93e640 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2023-10-20T03:33:43.617 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:43.611+0000 7fbd9d93e640 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2023-10-20T03:33:43.617 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:43.611+0000 7fbd9d93e640 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2023-10-20T03:33:43.618 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:43.615+0000 7fbd9d93e640 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2023-10-20T03:33:46.639 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2023-10-20T03:33:46.694 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:33:46.777 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:46.775+0000 7fb86969c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2023-10-20T03:33:46.778 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:46.775+0000 7fb86969c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2023-10-20T03:33:46.778 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:46.775+0000 7fb86969c640 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2023-10-20T03:33:46.778 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:46.775+0000 7fb86969c640 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2023-10-20T03:33:46.779 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:46.775+0000 7fb86969c640 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2023-10-20T03:33:46.779 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:33:46.775+0000 7fb86969c640 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2023-10-20T03:33:49.806 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2023-10-20T03:33:49.867 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-20T03:33:49.882 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-10-20T03:33:49.883 INFO:tasks.ceph:role: osd.1 2023-10-20T03:33:49.883 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-20T03:33:49.883 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-20T03:33:49.947 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:33:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:33:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:33:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:33:49.949 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:33:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:33:49.949 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:33:49.949 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:33:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:33:49.950 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:33:50.430 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-10-20T03:33:50.433 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-10-20T03:33:50.433 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2023-10-20T03:33:50.566 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-20T03:33:50.578 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:33:50.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:33:50.580 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-20T03:33:50.635 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-10-20T03:33:50.636 INFO:tasks.ceph:role: osd.5 2023-10-20T03:33:50.636 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-20T03:33:50.636 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-20T03:33:50.701 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:33:50.702 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:33:50.702 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:33:50.702 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:33:50.702 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:33:50.703 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:33:50.703 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:33:50.703 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:33:50.703 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:33:50.703 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:33:51.152 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-10-20T03:33:51.154 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-10-20T03:33:51.154 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-20T03:33:51.231 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-20T03:33:51.285 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:33:51.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:33:51.286 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2023-10-20T03:33:51.343 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-10-20T03:33:51.343 INFO:tasks.ceph:role: osd.9 2023-10-20T03:33:51.344 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-20T03:33:51.344 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-20T03:33:51.408 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:33:51.408 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:33:51.409 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:33:51.409 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:33:51.409 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:33:51.409 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:33:51.409 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:33:51.410 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:33:51.410 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:33:51.410 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:33:51.869 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-10-20T03:33:51.871 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-10-20T03:33:51.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2023-10-20T03:33:52.027 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2023-10-20T03:33:52.039 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:33:52.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:33:52.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2023-10-20T03:33:52.095 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-10-20T03:33:52.095 INFO:tasks.ceph:role: osd.13 2023-10-20T03:33:52.096 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-20T03:33:52.096 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-20T03:33:52.160 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:33:52.161 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:33:52.161 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:33:52.161 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:33:52.161 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:33:52.162 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:33:52.162 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:33:52.162 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:33:52.162 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:33:52.162 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:33:52.652 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-10-20T03:33:52.654 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-10-20T03:33:52.655 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2023-10-20T03:33:52.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2023-10-20T03:33:52.800 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:33:52.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:33:52.801 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:33:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:52.921+0000 7fbbe6d4a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-20T03:33:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:52.925+0000 7fbbe6d4a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-20T03:33:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:52.925+0000 7fbbe6d4a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-20T03:33:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:52.925+0000 7fbbe6d4a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-20T03:33:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:52.925+0000 7fbbe6d4a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-20T03:33:52.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:52.925+0000 7fbbe6d4a640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-20T03:33:55.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-20T03:33:55.965 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:33:56.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:56.045+0000 7f4558fe5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-20T03:33:56.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:56.045+0000 7f4558fe5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-20T03:33:56.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:56.045+0000 7f4558fe5640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-20T03:33:56.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:56.045+0000 7f4558fe5640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-20T03:33:56.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:56.045+0000 7f4558fe5640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-20T03:33:56.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:56.045+0000 7f4558fe5640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-20T03:33:59.064 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-20T03:33:59.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:33:59.202 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:59.197+0000 7ff7a01c3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2023-10-20T03:33:59.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:59.201+0000 7ff7a01c3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2023-10-20T03:33:59.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:59.201+0000 7ff7a01c3640 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2023-10-20T03:33:59.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:59.201+0000 7ff7a01c3640 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2023-10-20T03:33:59.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:59.201+0000 7ff7a01c3640 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2023-10-20T03:33:59.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:33:59.201+0000 7ff7a01c3640 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2023-10-20T03:34:02.223 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2023-10-20T03:34:02.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:34:02.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:34:02.361+0000 7fc9111e3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2023-10-20T03:34:02.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:34:02.361+0000 7fc9111e3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2023-10-20T03:34:02.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:34:02.361+0000 7fc9111e3640 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2023-10-20T03:34:02.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:34:02.361+0000 7fc9111e3640 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2023-10-20T03:34:02.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:34:02.361+0000 7fc9111e3640 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2023-10-20T03:34:02.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-20T03:34:02.361+0000 7fc9111e3640 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2023-10-20T03:34:05.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2023-10-20T03:34:05.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-20T03:34:05.471 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-10-20T03:34:05.471 INFO:tasks.ceph:role: osd.2 2023-10-20T03:34:05.471 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-20T03:34:05.472 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-20T03:34:05.536 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:34:05.536 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:34:05.536 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:34:05.536 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:34:05.536 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:34:05.537 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:34:05.537 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:34:05.537 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:34:05.537 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:34:05.537 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:34:06.004 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2023-10-20T03:34:06.007 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-10-20T03:34:06.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2023-10-20T03:34:06.163 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-20T03:34:06.174 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:34:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:34:06.176 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-20T03:34:06.231 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-10-20T03:34:06.232 INFO:tasks.ceph:role: osd.6 2023-10-20T03:34:06.232 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-20T03:34:06.232 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-20T03:34:06.299 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:34:06.299 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:34:06.300 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:34:06.300 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:34:06.300 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:34:06.300 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:34:06.301 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:34:06.301 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:34:06.301 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:34:06.301 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:34:06.751 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2023-10-20T03:34:06.753 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-10-20T03:34:06.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2023-10-20T03:34:06.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-20T03:34:06.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:34:06.895 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:34:06.896 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2023-10-20T03:34:06.950 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-10-20T03:34:06.951 INFO:tasks.ceph:role: osd.10 2023-10-20T03:34:06.951 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-20T03:34:06.951 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-20T03:34:07.012 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:34:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:34:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:34:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:34:07.013 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:34:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:34:07.013 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:34:07.013 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:34:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:34:07.014 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:34:07.469 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2023-10-20T03:34:07.471 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-10-20T03:34:07.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2023-10-20T03:34:07.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2023-10-20T03:34:07.606 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:34:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:34:07.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2023-10-20T03:34:07.668 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-10-20T03:34:07.668 INFO:tasks.ceph:role: osd.14 2023-10-20T03:34:07.668 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-20T03:34:07.669 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-20T03:34:07.732 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:34:07.732 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:34:07.732 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:34:07.732 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:34:07.733 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:34:07.733 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:34:07.733 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:34:07.733 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:34:07.733 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:34:07.734 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:34:08.226 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2023-10-20T03:34:08.228 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-10-20T03:34:08.228 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2023-10-20T03:34:08.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2023-10-20T03:34:08.383 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:34:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:34:08.384 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:34:08.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:08.502+0000 7ff2f0825640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-20T03:34:08.505 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:08.502+0000 7ff2f0825640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-20T03:34:08.505 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:08.502+0000 7ff2f0825640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-20T03:34:08.505 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:08.502+0000 7ff2f0825640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-20T03:34:08.505 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:08.502+0000 7ff2f0825640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-20T03:34:08.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:08.502+0000 7ff2f0825640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-20T03:34:11.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-20T03:34:11.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:34:11.608 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:11.606+0000 7f686275b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-20T03:34:11.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:11.606+0000 7f686275b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-20T03:34:11.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:11.606+0000 7f686275b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-20T03:34:11.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:11.606+0000 7f686275b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-20T03:34:11.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:11.606+0000 7f686275b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-20T03:34:11.610 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:11.606+0000 7f686275b640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-20T03:34:14.635 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-20T03:34:14.692 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:34:14.776 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:14.774+0000 7fdfbcd31640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2023-10-20T03:34:14.777 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:14.774+0000 7fdfbcd31640 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2023-10-20T03:34:14.777 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:14.774+0000 7fdfbcd31640 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2023-10-20T03:34:14.777 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:14.774+0000 7fdfbcd31640 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2023-10-20T03:34:14.777 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:14.774+0000 7fdfbcd31640 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2023-10-20T03:34:14.778 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:14.774+0000 7fdfbcd31640 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2023-10-20T03:34:17.784 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2023-10-20T03:34:17.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:34:17.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:17.922+0000 7ffbe29c8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2023-10-20T03:34:17.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:17.922+0000 7ffbe29c8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2023-10-20T03:34:17.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:17.922+0000 7ffbe29c8640 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2023-10-20T03:34:17.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:17.922+0000 7ffbe29c8640 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2023-10-20T03:34:17.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:17.922+0000 7ffbe29c8640 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2023-10-20T03:34:17.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-10-20T03:34:17.922+0000 7ffbe29c8640 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2023-10-20T03:34:20.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2023-10-20T03:34:21.016 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-20T03:34:21.030 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-10-20T03:34:21.030 INFO:tasks.ceph:role: osd.3 2023-10-20T03:34:21.031 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-20T03:34:21.031 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-20T03:34:21.089 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:34:21.089 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:34:21.090 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:34:21.090 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:34:21.090 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:34:21.090 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:34:21.091 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:34:21.091 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:34:21.091 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:34:21.091 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:34:21.113 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-20T03:34:21.277 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-20T03:34:21.277 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2023-10-20T03:34:21.336 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-20T03:34:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:34:21.389 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:34:21.390 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-20T03:34:21.441 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-10-20T03:34:21.442 INFO:tasks.ceph:role: osd.7 2023-10-20T03:34:21.442 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-20T03:34:21.443 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-20T03:34:21.497 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:34:21.498 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:34:21.498 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:34:21.498 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:34:21.498 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:34:21.499 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:34:21.499 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:34:21.499 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:34:21.499 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:34:21.499 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:34:21.521 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-20T03:34:21.523 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-20T03:34:21.524 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2023-10-20T03:34:21.581 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-20T03:34:21.630 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:34:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:34:21.632 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2023-10-20T03:34:21.686 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-10-20T03:34:21.687 INFO:tasks.ceph:role: osd.11 2023-10-20T03:34:21.687 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-20T03:34:21.687 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-20T03:34:21.742 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:34:21.743 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:34:21.743 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:34:21.743 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:34:21.743 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:34:21.744 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:34:21.744 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:34:21.744 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:34:21.744 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:34:21.744 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:34:21.766 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-20T03:34:21.769 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-20T03:34:21.769 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2023-10-20T03:34:21.826 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2023-10-20T03:34:21.878 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:34:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:34:21.880 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2023-10-20T03:34:21.932 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-10-20T03:34:21.932 INFO:tasks.ceph:role: osd.15 2023-10-20T03:34:21.932 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-20T03:34:21.933 DEBUG:teuthology.orchestra.run.smithi170:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-20T03:34:21.987 INFO:teuthology.orchestra.run.smithi170.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-20T03:34:21.987 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-20T03:34:21.987 INFO:teuthology.orchestra.run.smithi170.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-20T03:34:21.987 INFO:teuthology.orchestra.run.smithi170.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-20T03:34:21.987 INFO:teuthology.orchestra.run.smithi170.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-20T03:34:21.988 INFO:teuthology.orchestra.run.smithi170.stdout: = sunit=0 swidth=0 blks 2023-10-20T03:34:21.988 INFO:teuthology.orchestra.run.smithi170.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-20T03:34:21.988 INFO:teuthology.orchestra.run.smithi170.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-20T03:34:21.988 INFO:teuthology.orchestra.run.smithi170.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-20T03:34:21.988 INFO:teuthology.orchestra.run.smithi170.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-20T03:34:22.010 INFO:teuthology.orchestra.run.smithi170.stdout:Discarding blocks...Done. 2023-10-20T03:34:22.012 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi170.front.sepia.ceph.com -o noatime 2023-10-20T03:34:22.012 DEBUG:teuthology.orchestra.run.smithi170:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2023-10-20T03:34:22.070 DEBUG:teuthology.orchestra.run.smithi170:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2023-10-20T03:34:22.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:34:22.124 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: /sbin/restorecon: command not found 2023-10-20T03:34:22.125 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:34:22.234 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:22.232+0000 7f90bbccc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-20T03:34:22.235 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:22.232+0000 7f90bbccc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-20T03:34:22.235 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:22.232+0000 7f90bbccc640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-20T03:34:22.235 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:22.232+0000 7f90bbccc640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-20T03:34:22.235 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:22.232+0000 7f90bbccc640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-20T03:34:22.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:22.232+0000 7f90bbccc640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-20T03:34:25.239 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-20T03:34:25.253 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:34:25.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:25.328+0000 7f1da1d37640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-20T03:34:25.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:25.328+0000 7f1da1d37640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-20T03:34:25.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:25.328+0000 7f1da1d37640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-20T03:34:25.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:25.328+0000 7f1da1d37640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-20T03:34:25.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:25.328+0000 7f1da1d37640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-20T03:34:25.335 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:25.328+0000 7f1da1d37640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-20T03:34:28.332 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-20T03:34:28.386 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:34:28.474 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:28.468+0000 7f5b69e08640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2023-10-20T03:34:28.474 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:28.468+0000 7f5b69e08640 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2023-10-20T03:34:28.475 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:28.468+0000 7f5b69e08640 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2023-10-20T03:34:28.475 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:28.468+0000 7f5b69e08640 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2023-10-20T03:34:28.475 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:28.472+0000 7f5b69e08640 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2023-10-20T03:34:28.475 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:28.472+0000 7f5b69e08640 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2023-10-20T03:34:31.475 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2023-10-20T03:34:31.530 DEBUG:teuthology.orchestra.run.smithi170:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:34:31.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:31.612+0000 7fb38ba05640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2023-10-20T03:34:31.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:31.612+0000 7fb38ba05640 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2023-10-20T03:34:31.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:31.612+0000 7fb38ba05640 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2023-10-20T03:34:31.618 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:31.612+0000 7fb38ba05640 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2023-10-20T03:34:31.618 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:31.612+0000 7fb38ba05640 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2023-10-20T03:34:31.618 INFO:teuthology.orchestra.run.smithi170.stderr:2023-10-20T03:34:31.612+0000 7fb38ba05640 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2023-10-20T03:34:34.621 DEBUG:teuthology.orchestra.run.smithi170:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2023-10-20T03:34:34.674 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-20T03:34:34.674 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:34:34.675 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-20T03:34:34.689 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:34:34.691 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-20T03:34:34.745 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:34:34.746 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-20T03:34:34.805 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:34:34.806 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2023-10-20T03:34:34.860 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:34:34.861 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2023-10-20T03:34:34.917 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:34:34.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-20T03:34:34.932 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:34:34.932 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-20T03:34:34.987 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:34:34.987 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2023-10-20T03:34:35.043 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:34:35.043 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2023-10-20T03:34:35.099 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:34:35.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-20T03:34:35.114 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:34:35.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-20T03:34:35.167 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:34:35.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2023-10-20T03:34:35.222 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:34:35.223 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2023-10-20T03:34:35.279 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:34:35.279 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-20T03:34:35.293 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:34:35.293 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-20T03:34:35.346 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:34:35.346 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-20T03:34:35.402 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:34:35.402 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2023-10-20T03:34:35.457 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:34:35.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2023-10-20T03:34:35.510 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:34:35.510 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-20T03:34:35.558 INFO:tasks.ceph:Adding keys to all mons... 2023-10-20T03:34:35.559 DEBUG:teuthology.orchestra.run.smithi033:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-20T03:34:35.561 DEBUG:teuthology.orchestra.run.smithi040:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-20T03:34:35.563 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-20T03:34:35.571 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.y] 2023-10-20T03:34:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAO9TFljQn3EhAAkKL84tkckyADp+zGVhY7Cg== 2023-10-20T03:34:35.572 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.0] 2023-10-20T03:34:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAR9TFl3PjUFBAAl8TkQNDRtPd6ME1n5J17kw== 2023-10-20T03:34:35.572 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.4] 2023-10-20T03:34:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAU9TFlETRnHBAAvRu6rBi7erycFpnYG4tORg== 2023-10-20T03:34:35.572 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.8] 2023-10-20T03:34:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAX9TFlmKjKJBAAeM0fwHz4LK5qnNY3PbWGTw== 2023-10-20T03:34:35.573 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.12] 2023-10-20T03:34:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAa9TFlKvRsLhAAo7dp3dW2omuYITvDF5LYSQ== 2023-10-20T03:34:35.573 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.1] 2023-10-20T03:34:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAg9TFloSAONxAAUcTRSYaoGqX8V4iwm1i/og== 2023-10-20T03:34:35.574 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.5] 2023-10-20T03:34:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAk9TFlnCjbAhAA2HNAaF6rijGv4dIdGa7OKQ== 2023-10-20T03:34:35.574 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.9] 2023-10-20T03:34:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAn9TFltlUSDBAAJ5ENa7ZV4CSanvf4ovq1AA== 2023-10-20T03:34:35.574 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.13] 2023-10-20T03:34:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAq9TFlpb2zFRAA8OEPrdojGO6H32U0u967tg== 2023-10-20T03:34:35.575 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.2] 2023-10-20T03:34:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAw9TFl2NIVHhAA/IQCIopv7YHLG2TfZEeUQQ== 2023-10-20T03:34:35.575 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.6] 2023-10-20T03:34:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAz9TFl21ZRJBAAuAw9Equ49zaFyHyUFDVv1g== 2023-10-20T03:34:35.576 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.10] 2023-10-20T03:34:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQA29TFlE81cLhAAOJOy6Xsb0rcTHCO6iKXTkw== 2023-10-20T03:34:35.576 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.14] 2023-10-20T03:34:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQA59TFlQEE/NxAANY0/0c8+dxAzRSHErvlKgQ== 2023-10-20T03:34:35.576 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.x] 2023-10-20T03:34:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAO9TFlX0mfFRAAiOZjwS96C4wb6ueNudn1Kw== 2023-10-20T03:34:35.577 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.3] 2023-10-20T03:34:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQA+9TFlCJUVDhAAz+LiRZarzLApXfgc3JYb4A== 2023-10-20T03:34:35.577 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.7] 2023-10-20T03:34:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBB9TFlQT/6ExAA7zHn8HAdqaq1ldFSc5A/AQ== 2023-10-20T03:34:35.577 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.11] 2023-10-20T03:34:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBE9TFl39hdHBAA2nwDBj0s/SUil0eL3n8opQ== 2023-10-20T03:34:35.578 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.15] 2023-10-20T03:34:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBH9TFlWO7dJBAAE00wNAVxCA4FUFeIRppmgQ== 2023-10-20T03:34:35.578 INFO:teuthology.orchestra.run.smithi033.stdout:[client.0] 2023-10-20T03:34:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAO9TFlPd/4GhAAC2KaPg1kODOWQ2HcZpNCpg== 2023-10-20T03:34:35.580 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.y] 2023-10-20T03:34:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAO9TFljQn3EhAAkKL84tkckyADp+zGVhY7Cg== 2023-10-20T03:34:35.581 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2023-10-20T03:34:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAR9TFl3PjUFBAAl8TkQNDRtPd6ME1n5J17kw== 2023-10-20T03:34:35.581 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.4] 2023-10-20T03:34:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAU9TFlETRnHBAAvRu6rBi7erycFpnYG4tORg== 2023-10-20T03:34:35.581 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.8] 2023-10-20T03:34:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAX9TFlmKjKJBAAeM0fwHz4LK5qnNY3PbWGTw== 2023-10-20T03:34:35.582 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.12] 2023-10-20T03:34:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAa9TFlKvRsLhAAo7dp3dW2omuYITvDF5LYSQ== 2023-10-20T03:34:35.582 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2023-10-20T03:34:35.582 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAg9TFloSAONxAAUcTRSYaoGqX8V4iwm1i/og== 2023-10-20T03:34:35.582 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.5] 2023-10-20T03:34:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAk9TFlnCjbAhAA2HNAaF6rijGv4dIdGa7OKQ== 2023-10-20T03:34:35.583 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.9] 2023-10-20T03:34:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAn9TFltlUSDBAAJ5ENa7ZV4CSanvf4ovq1AA== 2023-10-20T03:34:35.583 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.13] 2023-10-20T03:34:35.583 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAq9TFlpb2zFRAA8OEPrdojGO6H32U0u967tg== 2023-10-20T03:34:35.583 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2023-10-20T03:34:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAw9TFl2NIVHhAA/IQCIopv7YHLG2TfZEeUQQ== 2023-10-20T03:34:35.584 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.6] 2023-10-20T03:34:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAz9TFl21ZRJBAAuAw9Equ49zaFyHyUFDVv1g== 2023-10-20T03:34:35.584 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.10] 2023-10-20T03:34:35.584 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQA29TFlE81cLhAAOJOy6Xsb0rcTHCO6iKXTkw== 2023-10-20T03:34:35.585 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.14] 2023-10-20T03:34:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQA59TFlQEE/NxAANY0/0c8+dxAzRSHErvlKgQ== 2023-10-20T03:34:35.585 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2023-10-20T03:34:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAO9TFlX0mfFRAAiOZjwS96C4wb6ueNudn1Kw== 2023-10-20T03:34:35.585 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2023-10-20T03:34:35.585 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQA+9TFlCJUVDhAAz+LiRZarzLApXfgc3JYb4A== 2023-10-20T03:34:35.586 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.7] 2023-10-20T03:34:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBB9TFlQT/6ExAA7zHn8HAdqaq1ldFSc5A/AQ== 2023-10-20T03:34:35.586 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.11] 2023-10-20T03:34:35.586 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBE9TFl39hdHBAA2nwDBj0s/SUil0eL3n8opQ== 2023-10-20T03:34:35.586 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.15] 2023-10-20T03:34:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQBH9TFlWO7dJBAAE00wNAVxCA4FUFeIRppmgQ== 2023-10-20T03:34:35.587 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2023-10-20T03:34:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAO9TFlPd/4GhAAC2KaPg1kODOWQ2HcZpNCpg== 2023-10-20T03:34:35.587 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.y] 2023-10-20T03:34:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAO9TFljQn3EhAAkKL84tkckyADp+zGVhY7Cg== 2023-10-20T03:34:35.588 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.0] 2023-10-20T03:34:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAR9TFl3PjUFBAAl8TkQNDRtPd6ME1n5J17kw== 2023-10-20T03:34:35.588 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.4] 2023-10-20T03:34:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAU9TFlETRnHBAAvRu6rBi7erycFpnYG4tORg== 2023-10-20T03:34:35.588 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.8] 2023-10-20T03:34:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAX9TFlmKjKJBAAeM0fwHz4LK5qnNY3PbWGTw== 2023-10-20T03:34:35.589 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.12] 2023-10-20T03:34:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAa9TFlKvRsLhAAo7dp3dW2omuYITvDF5LYSQ== 2023-10-20T03:34:35.589 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.1] 2023-10-20T03:34:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAg9TFloSAONxAAUcTRSYaoGqX8V4iwm1i/og== 2023-10-20T03:34:35.589 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.5] 2023-10-20T03:34:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAk9TFlnCjbAhAA2HNAaF6rijGv4dIdGa7OKQ== 2023-10-20T03:34:35.590 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.9] 2023-10-20T03:34:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAn9TFltlUSDBAAJ5ENa7ZV4CSanvf4ovq1AA== 2023-10-20T03:34:35.590 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.13] 2023-10-20T03:34:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAq9TFlpb2zFRAA8OEPrdojGO6H32U0u967tg== 2023-10-20T03:34:35.590 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.2] 2023-10-20T03:34:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAw9TFl2NIVHhAA/IQCIopv7YHLG2TfZEeUQQ== 2023-10-20T03:34:35.591 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.6] 2023-10-20T03:34:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAz9TFl21ZRJBAAuAw9Equ49zaFyHyUFDVv1g== 2023-10-20T03:34:35.591 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.10] 2023-10-20T03:34:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQA29TFlE81cLhAAOJOy6Xsb0rcTHCO6iKXTkw== 2023-10-20T03:34:35.592 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.14] 2023-10-20T03:34:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQA59TFlQEE/NxAANY0/0c8+dxAzRSHErvlKgQ== 2023-10-20T03:34:35.592 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.x] 2023-10-20T03:34:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAO9TFlX0mfFRAAiOZjwS96C4wb6ueNudn1Kw== 2023-10-20T03:34:35.592 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.3] 2023-10-20T03:34:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQA+9TFlCJUVDhAAz+LiRZarzLApXfgc3JYb4A== 2023-10-20T03:34:35.593 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.7] 2023-10-20T03:34:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBB9TFlQT/6ExAA7zHn8HAdqaq1ldFSc5A/AQ== 2023-10-20T03:34:35.593 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.11] 2023-10-20T03:34:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBE9TFl39hdHBAA2nwDBj0s/SUil0eL3n8opQ== 2023-10-20T03:34:35.593 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.15] 2023-10-20T03:34:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBH9TFlWO7dJBAAE00wNAVxCA4FUFeIRppmgQ== 2023-10-20T03:34:35.594 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2023-10-20T03:34:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAO9TFlPd/4GhAAC2KaPg1kODOWQ2HcZpNCpg== 2023-10-20T03:34:35.594 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-20T03:34:35.626 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-20T03:34:35.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-20T03:34:35.676 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:35.678 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:35.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:35.761 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:35.763 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:35.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:35.844 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:35.847 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:35.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:35.929 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:35.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:35.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.015 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.060 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.102 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.105 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.187 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.190 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.228 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.272 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.358 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.361 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.400 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.449 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.451 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.454 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.495 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.538 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.583 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.624 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.675 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-20T03:34:36.710 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-20T03:34:36.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-20T03:34:36.765 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.794 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.848 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.878 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.881 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.937 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.962 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:36.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:37.009 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:37.046 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:37.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-20T03:34:37.093 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-20T03:34:37.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-20T03:34:37.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-20T03:34:37.180 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-20T03:34:37.181 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-20T03:34:37.193 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-20T03:34:37.294 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-20T03:34:37.306 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-20T03:34:37.321 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-20T03:34:37.424 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-20T03:34:37.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-20T03:34:37.450 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-20T03:34:37.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-20T03:34:37.564 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:34:37.566 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:34:37.569 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-20T03:34:37.611 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-20T03:34:37.612 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-20T03:34:37.612 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-20T03:34:37.615 INFO:tasks.ceph.mon.a:Started 2023-10-20T03:34:37.615 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-20T03:34:37.616 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-20T03:34:37.618 INFO:tasks.ceph.mon.b:Started 2023-10-20T03:34:37.619 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-20T03:34:37.619 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-20T03:34:37.656 INFO:tasks.ceph.mon.c:Started 2023-10-20T03:34:37.656 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-20T03:34:37.657 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-20T03:34:37.657 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-20T03:34:37.660 INFO:tasks.ceph.mgr.y:Started 2023-10-20T03:34:37.660 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-20T03:34:37.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-20T03:34:37.663 INFO:tasks.ceph.mgr.x:Started 2023-10-20T03:34:37.663 INFO:tasks.ceph:Setting crush tunables to default 2023-10-20T03:34:37.664 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd crush tunables default 2023-10-20T03:34:48.236 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:48.232+0000 7f6645f86180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-20T03:34:48.365 INFO:teuthology.orchestra.run.smithi033.stderr:adjusted tunables profile to default 2023-10-20T03:34:48.384 INFO:tasks.ceph:check_enable_crimson: False 2023-10-20T03:34:48.384 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-20T03:34:48.385 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:34:48.385 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-20T03:34:48.397 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:34:48.398 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-20T03:34:48.409 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:48.407+0000 7f589fe24180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-20T03:34:48.420 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:34:48.420 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2023-10-20T03:34:48.471 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-10-20T03:34:48.471 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2023-10-20T03:34:48.524 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:34:48.525 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-20T03:34:48.539 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:34:48.540 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-20T03:34:48.595 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:34:48.595 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2023-10-20T03:34:48.654 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-20T03:34:48.655 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2023-10-20T03:34:48.710 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:34:48.710 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-20T03:34:48.725 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:34:48.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-20T03:34:48.779 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:34:48.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2023-10-20T03:34:48.834 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-10-20T03:34:48.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2023-10-20T03:34:48.855 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:48.852+0000 7f6645f86180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-20T03:34:48.891 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:34:48.891 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-20T03:34:48.904 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:34:48.905 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-20T03:34:48.955 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:48.951+0000 7f589fe24180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-20T03:34:48.957 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:34:48.957 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2023-10-20T03:34:49.009 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-10-20T03:34:49.010 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2023-10-20T03:34:49.028 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:49.024+0000 7f6645f86180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-20T03:34:49.039 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 4872451c-1bc7-4f21-a95f-986fa428ea65 0 2023-10-20T03:34:49.154 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:49.151+0000 7f589fe24180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-20T03:34:49.280 INFO:teuthology.orchestra.run.smithi170.stdout:0 2023-10-20T03:34:49.296 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 0b570426-e165-4caa-a914-e1bed7d805fc 1 2023-10-20T03:34:49.490 INFO:teuthology.orchestra.run.smithi170.stdout:1 2023-10-20T03:34:49.508 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new e0184c10-844b-4bcb-adaf-a0d407b708ee 2 2023-10-20T03:34:49.603 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:49.600+0000 7f6645f86180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-20T03:34:49.660 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:49.656+0000 7f6645f86180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-20T03:34:49.706 INFO:teuthology.orchestra.run.smithi170.stdout:2 2023-10-20T03:34:49.714 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:49.712+0000 7f6645f86180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-20T03:34:49.723 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 8120023b-1d48-411d-a642-2529a697b5c9 3 2023-10-20T03:34:49.750 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:49.747+0000 7f589fe24180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-20T03:34:49.795 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:49.792+0000 7f6645f86180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-20T03:34:49.806 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:49.803+0000 7f589fe24180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-20T03:34:49.850 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:49.848+0000 7f6645f86180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-20T03:34:49.862 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:49.859+0000 7f589fe24180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-20T03:34:49.922 INFO:teuthology.orchestra.run.smithi170.stdout:3 2023-10-20T03:34:49.929 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:49.924+0000 7f6645f86180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-20T03:34:49.939 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 549b3b39-41d5-4d9b-932e-a7b3e8cc0684 4 2023-10-20T03:34:49.943 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:49.939+0000 7f589fe24180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-20T03:34:49.983 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:49.980+0000 7f6645f86180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-20T03:34:49.998 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:49.995+0000 7f589fe24180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-20T03:34:50.079 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:50.075+0000 7f589fe24180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-20T03:34:50.129 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:50.124+0000 7f6645f86180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-20T03:34:50.133 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:50.131+0000 7f589fe24180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-20T03:34:50.136 INFO:teuthology.orchestra.run.smithi170.stdout:4 2023-10-20T03:34:50.152 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new a99ea3db-6433-4312-83d1-64ba69f028a4 5 2023-10-20T03:34:50.193 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:50.188+0000 7f6645f86180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-20T03:34:50.279 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:50.275+0000 7f589fe24180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-20T03:34:50.341 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:50.339+0000 7f589fe24180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-20T03:34:50.364 INFO:teuthology.orchestra.run.smithi170.stdout:5 2023-10-20T03:34:50.380 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new d2b38063-49eb-4f56-bb8d-411419b71b9b 6 2023-10-20T03:34:50.654 INFO:teuthology.orchestra.run.smithi170.stdout:6 2023-10-20T03:34:50.670 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 80e34f64-35dc-4a6d-9f7f-c2263c78350d 7 2023-10-20T03:34:50.790 INFO:tasks.ceph.mgr.x.smithi170.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-20T03:34:50.790 INFO:tasks.ceph.mgr.x.smithi170.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-20T03:34:50.790 INFO:tasks.ceph.mgr.x.smithi170.stderr: from numpy import show_config as show_numpy_config 2023-10-20T03:34:50.842 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:50.840+0000 7f6645f86180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-20T03:34:50.861 INFO:teuthology.orchestra.run.smithi170.stdout:7 2023-10-20T03:34:50.877 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 1cd10625-66a8-4488-b1e5-378cb0062107 8 2023-10-20T03:34:50.968 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:50.964+0000 7f6645f86180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-20T03:34:51.013 INFO:tasks.ceph.mgr.y.smithi033.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-20T03:34:51.014 INFO:tasks.ceph.mgr.y.smithi033.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-20T03:34:51.014 INFO:tasks.ceph.mgr.y.smithi033.stderr: from numpy import show_config as show_numpy_config 2023-10-20T03:34:51.074 INFO:teuthology.orchestra.run.smithi170.stdout:8 2023-10-20T03:34:51.079 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:51.075+0000 7f589fe24180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-20T03:34:51.091 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4 9 2023-10-20T03:34:51.132 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:51.128+0000 7f6645f86180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-20T03:34:51.198 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:51.196+0000 7f6645f86180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-20T03:34:51.249 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:51.247+0000 7f589fe24180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-20T03:34:51.301 INFO:teuthology.orchestra.run.smithi170.stdout:9 2023-10-20T03:34:51.317 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 42a67bcc-5c07-499d-8902-08c2f15e7fbb 10 2023-10-20T03:34:51.340 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:51.336+0000 7f6645f86180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-20T03:34:51.423 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:51.419+0000 7f589fe24180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-20T03:34:51.494 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:51.491+0000 7f589fe24180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-20T03:34:51.542 INFO:teuthology.orchestra.run.smithi170.stdout:10 2023-10-20T03:34:51.558 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new ab1db647-aada-4deb-abdc-0a1baff548db 11 2023-10-20T03:34:51.632 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:51.627+0000 7f589fe24180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-20T03:34:51.757 INFO:teuthology.orchestra.run.smithi170.stdout:11 2023-10-20T03:34:51.775 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new b20cee74-2cce-4153-97c2-56e9db40c712 12 2023-10-20T03:34:51.962 INFO:teuthology.orchestra.run.smithi170.stdout:12 2023-10-20T03:34:51.975 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:51.972+0000 7f6645f86180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-20T03:34:51.979 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 0974cd04-c690-4cba-ac05-5d404493dd1d 13 2023-10-20T03:34:52.054 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:52.052+0000 7f6645f86180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-20T03:34:52.180 INFO:teuthology.orchestra.run.smithi170.stdout:13 2023-10-20T03:34:52.197 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new 3bf53124-13b6-4e5c-9868-2382719ffe25 14 2023-10-20T03:34:52.233 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:52.228+0000 7f6645f86180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-20T03:34:52.274 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:52.271+0000 7f589fe24180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-20T03:34:52.309 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:52.304+0000 7f6645f86180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-20T03:34:52.352 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:52.347+0000 7f589fe24180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-20T03:34:52.395 INFO:teuthology.orchestra.run.smithi170.stdout:14 2023-10-20T03:34:52.411 DEBUG:teuthology.orchestra.run.smithi170:> sudo ceph --cluster ceph osd new b0b5ffe5-a25b-412f-818f-b60907c0089b 15 2023-10-20T03:34:52.531 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:52.527+0000 7f589fe24180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-20T03:34:52.606 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:52.604+0000 7f6645f86180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-20T03:34:52.608 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:52.603+0000 7f589fe24180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-20T03:34:52.645 INFO:teuthology.orchestra.run.smithi170.stdout:15 2023-10-20T03:34:52.661 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-20T03:34:52.662 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-20T03:34:52.665 INFO:tasks.ceph.osd.0:Started 2023-10-20T03:34:52.665 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-20T03:34:52.666 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-20T03:34:52.667 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:52.664+0000 7f6645f86180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-20T03:34:52.668 INFO:tasks.ceph.osd.4:Started 2023-10-20T03:34:52.669 INFO:tasks.ceph.osd.8:Restarting daemon 2023-10-20T03:34:52.669 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2023-10-20T03:34:52.672 INFO:tasks.ceph.osd.8:Started 2023-10-20T03:34:52.672 INFO:tasks.ceph.osd.12:Restarting daemon 2023-10-20T03:34:52.673 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2023-10-20T03:34:52.675 INFO:tasks.ceph.osd.12:Started 2023-10-20T03:34:52.676 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-20T03:34:52.676 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-20T03:34:52.679 INFO:tasks.ceph.osd.1:Started 2023-10-20T03:34:52.679 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-20T03:34:52.680 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-20T03:34:52.682 INFO:tasks.ceph.osd.5:Started 2023-10-20T03:34:52.683 INFO:tasks.ceph.osd.9:Restarting daemon 2023-10-20T03:34:52.683 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2023-10-20T03:34:52.686 INFO:tasks.ceph.osd.9:Started 2023-10-20T03:34:52.686 INFO:tasks.ceph.osd.13:Restarting daemon 2023-10-20T03:34:52.687 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2023-10-20T03:34:52.689 INFO:tasks.ceph.osd.13:Started 2023-10-20T03:34:52.691 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-20T03:34:52.692 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-20T03:34:52.694 INFO:tasks.ceph.osd.2:Started 2023-10-20T03:34:52.695 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-20T03:34:52.695 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-20T03:34:52.698 INFO:tasks.ceph.osd.6:Started 2023-10-20T03:34:52.698 INFO:tasks.ceph.osd.10:Restarting daemon 2023-10-20T03:34:52.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2023-10-20T03:34:52.701 INFO:tasks.ceph.osd.10:Started 2023-10-20T03:34:52.702 INFO:tasks.ceph.osd.14:Restarting daemon 2023-10-20T03:34:52.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2023-10-20T03:34:52.705 INFO:tasks.ceph.osd.14:Started 2023-10-20T03:34:52.705 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-20T03:34:52.706 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-20T03:34:52.708 INFO:tasks.ceph.osd.3:Started 2023-10-20T03:34:52.708 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-20T03:34:52.709 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-20T03:34:52.711 INFO:tasks.ceph.osd.7:Started 2023-10-20T03:34:52.711 INFO:tasks.ceph.osd.11:Restarting daemon 2023-10-20T03:34:52.711 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2023-10-20T03:34:52.713 INFO:tasks.ceph.osd.11:Started 2023-10-20T03:34:52.714 INFO:tasks.ceph.osd.15:Restarting daemon 2023-10-20T03:34:52.714 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2023-10-20T03:34:52.717 INFO:tasks.ceph.osd.15:Started 2023-10-20T03:34:52.718 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:34:52.912 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:34:52.912 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":18,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:34:52.643595+0000","last_up_change":"0.000000","last_in_change":"2023-10-20T03:34:52.643595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:34:52.930 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-20T03:34:52.931 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-20T03:34:52.950 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:52.947+0000 7f589fe24180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-20T03:34:53.013 INFO:tasks.ceph.mgr.y.smithi033.stderr:2023-10-20T03:34:53.011+0000 7f589fe24180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-20T03:34:53.296 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:34:53.293+0000 7ff0bfbb8640 -1 Falling back to public interface 2023-10-20T03:34:53.303 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:34:53.299+0000 7f5264bef640 -1 Falling back to public interface 2023-10-20T03:34:53.311 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:34:53.308+0000 7f541ccc8640 -1 Falling back to public interface 2023-10-20T03:34:53.315 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:34:53.313+0000 7f1072d78640 -1 Falling back to public interface 2023-10-20T03:34:53.316 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:34:53.313+0000 7f47dde08640 -1 Falling back to public interface 2023-10-20T03:34:53.316 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:34:53.313+0000 7f230b908640 -1 Falling back to public interface 2023-10-20T03:34:53.317 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:34:53.315+0000 7fa85950b640 -1 Falling back to public interface 2023-10-20T03:34:53.321 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:34:53.318+0000 7fd161fdc640 -1 Falling back to public interface 2023-10-20T03:34:53.322 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:34:53.319+0000 7f1816db5640 -1 Falling back to public interface 2023-10-20T03:34:53.324 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:34:53.322+0000 7fc9d3d3c640 -1 Falling back to public interface 2023-10-20T03:34:53.325 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:34:53.323+0000 7f8064da2640 -1 Falling back to public interface 2023-10-20T03:34:53.331 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:34:53.328+0000 7fe4d1816640 -1 Falling back to public interface 2023-10-20T03:34:53.332 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:34:53.330+0000 7f201f7b6640 -1 Falling back to public interface 2023-10-20T03:34:53.343 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:34:53.340+0000 7f306bc29640 -1 Falling back to public interface 2023-10-20T03:34:53.343 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:34:53.340+0000 7f9e57b7a640 -1 Falling back to public interface 2023-10-20T03:34:53.348 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:34:53.346+0000 7f80e4254640 -1 Falling back to public interface 2023-10-20T03:34:53.943 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:53.940+0000 7f65ed7fa640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-20T03:34:53.947 INFO:tasks.ceph.mgr.x.smithi170.stderr:2023-10-20T03:34:53.944+0000 7f65ed7fa640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-20T03:34:54.545 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-20T03:34:54.840 INFO:teuthology.misc.health.smithi033.stdout: 2023-10-20T03:34:54.840 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":18,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:34:52.643595+0000","last_up_change":"0.000000","last_in_change":"2023-10-20T03:34:52.643595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:34:54.857 DEBUG:teuthology.misc:0 of 16 OSDs are up 2023-10-20T03:34:56.905 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:34:56.901+0000 7f47dde08640 -1 osd.5 0 log_to_monitors true 2023-10-20T03:34:57.010 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:34:57.005+0000 7ff0bfbb8640 -1 osd.9 0 log_to_monitors true 2023-10-20T03:34:57.071 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:34:57.072+0000 7f541ccc8640 -1 osd.11 0 log_to_monitors true 2023-10-20T03:34:57.085 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:34:57.082+0000 7f5264bef640 -1 osd.8 0 log_to_monitors true 2023-10-20T03:34:57.091 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:34:57.086+0000 7f1816db5640 -1 osd.12 0 log_to_monitors true 2023-10-20T03:34:57.133 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:34:57.129+0000 7f230b908640 -1 osd.13 0 log_to_monitors true 2023-10-20T03:34:57.139 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:34:57.137+0000 7f1072d78640 -1 osd.1 0 log_to_monitors true 2023-10-20T03:34:57.190 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:34:57.188+0000 7fe4d1816640 -1 osd.7 0 log_to_monitors true 2023-10-20T03:34:57.191 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:34:57.192+0000 7f306bc29640 -1 osd.3 0 log_to_monitors true 2023-10-20T03:34:57.205 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:34:57.202+0000 7f8064da2640 -1 osd.0 0 log_to_monitors true 2023-10-20T03:34:57.206 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:34:57.201+0000 7fd161fdc640 -1 osd.10 0 log_to_monitors true 2023-10-20T03:34:57.213 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:34:57.210+0000 7fa85950b640 -1 osd.4 0 log_to_monitors true 2023-10-20T03:34:57.261 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:34:57.257+0000 7f80e4254640 -1 osd.2 0 log_to_monitors true 2023-10-20T03:34:57.267 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:34:57.265+0000 7f201f7b6640 -1 osd.14 0 log_to_monitors true 2023-10-20T03:34:57.277 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:34:57.273+0000 7fc9d3d3c640 -1 osd.6 0 log_to_monitors true 2023-10-20T03:34:57.286 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:34:57.284+0000 7f9e57b7a640 -1 osd.15 0 log_to_monitors true 2023-10-20T03:34:57.990 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:34:57.989+0000 7f47d9db0640 -1 osd.5 0 waiting for initial osdmap 2023-10-20T03:34:58.023 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:34:58.021+0000 7f47d23e6640 -1 osd.5 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:58.987 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:34:58.981+0000 7fd15df84640 -1 osd.10 0 waiting for initial osdmap 2023-10-20T03:34:58.988 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:34:58.985+0000 7ff0bbb60640 -1 osd.9 0 waiting for initial osdmap 2023-10-20T03:34:58.989 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:34:58.985+0000 7fc9cfce4640 -1 osd.6 0 waiting for initial osdmap 2023-10-20T03:34:58.989 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:34:58.988+0000 7f5418c70640 -1 osd.11 0 waiting for initial osdmap 2023-10-20T03:34:58.992 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:34:58.992+0000 7f9e53b22640 -1 osd.15 0 waiting for initial osdmap 2023-10-20T03:34:59.000 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:34:58.997+0000 7f106ed20640 -1 osd.1 0 waiting for initial osdmap 2023-10-20T03:34:59.000 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:34:58.997+0000 7f23078b0640 -1 osd.13 0 waiting for initial osdmap 2023-10-20T03:34:59.002 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:34:58.998+0000 7f5260b97640 -1 osd.8 0 waiting for initial osdmap 2023-10-20T03:34:59.003 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:34:58.998+0000 7f1812d5d640 -1 osd.12 0 waiting for initial osdmap 2023-10-20T03:34:59.003 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:34:58.998+0000 7f8060d4a640 -1 osd.0 0 waiting for initial osdmap 2023-10-20T03:34:59.003 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:34:58.998+0000 7fa8554b3640 -1 osd.4 0 waiting for initial osdmap 2023-10-20T03:34:59.004 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:34:58.997+0000 7f201b75e640 -1 osd.14 0 waiting for initial osdmap 2023-10-20T03:34:59.004 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:34:59.004+0000 7f3067bd1640 -1 osd.3 0 waiting for initial osdmap 2023-10-20T03:34:59.005 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:34:58.997+0000 7f80e01fc640 -1 osd.2 0 waiting for initial osdmap 2023-10-20T03:34:59.005 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:34:59.004+0000 7fe4cd7be640 -1 osd.7 0 waiting for initial osdmap 2023-10-20T03:34:59.019 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:34:59.017+0000 7ff0b4196640 -1 osd.9 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.027 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:34:59.025+0000 7fd156dbb640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.035 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:34:59.036+0000 7f3060a08640 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.035 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:34:59.036+0000 7f54112a6640 -1 osd.11 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.036 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:34:59.036+0000 7f9e4c959640 -1 osd.15 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.036 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:34:59.033+0000 7f80d9033640 -1 osd.2 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.036 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:34:59.033+0000 7fc9c8b1b640 -1 osd.6 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.039 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:34:59.037+0000 7f1067356640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.046 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:34:59.042+0000 7f52591cd640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.046 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:34:59.042+0000 7f8059b81640 -1 osd.0 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.046 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:34:59.042+0000 7f180b393640 -1 osd.12 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.048 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:34:59.045+0000 7f2014595640 -1 osd.14 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.055 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:34:59.053+0000 7f22ffee6640 -1 osd.13 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.057 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:34:59.054+0000 7fa84e2ea640 -1 osd.4 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:34:59.058 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:34:59.060+0000 7fe4c65f5640 -1 osd.7 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:35:02.473 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-20T03:35:02.773 INFO:teuthology.misc.health.smithi033.stdout: 2023-10-20T03:35:02.774 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":24,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:35:02.017685+0000","last_up_change":"2023-10-20T03:34:59.991660+0000","last_in_change":"2023-10-20T03:34:52.643595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2603014448},{"type":"v1","addr":"172.21.15.64:6809","nonce":2603014448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2603014448},{"type":"v1","addr":"172.21.15.64:6811","nonce":2603014448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2603014448},{"type":"v1","addr":"172.21.15.64:6815","nonce":2603014448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2603014448},{"type":"v1","addr":"172.21.15.64:6813","nonce":2603014448}]},"public_addr":"172.21.15.64:6809/2603014448","cluster_addr":"172.21.15.64:6811/2603014448","heartbeat_back_addr":"172.21.15.64:6815/2603014448","heartbeat_front_addr":"172.21.15.64:6813/2603014448","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists","up"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.204996+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.164818+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.304156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.175698+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.169661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:57.863151+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.279628+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.238525+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.074220+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.020311+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.177070+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.063374+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.069832+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.130292+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.265757+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.317166+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:35:02.791 DEBUG:teuthology.misc:16 of 16 OSDs are up 2023-10-20T03:35:02.792 INFO:tasks.ceph:Creating RBD pool 2023-10-20T03:35:02.792 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-20T03:35:04.082 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'rbd' created 2023-10-20T03:35:04.105 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-20T03:35:05.094 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-20T03:35:05.124 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-20T03:35:05.125 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-20T03:35:05.125 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-20T03:35:05.125 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-20T03:35:05.126 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-20T03:35:05.467 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:35:05.486 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":6,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":2433496486},{"type":"v1","addr":"172.21.15.170:6801","nonce":2433496486}]},"active_addr":"172.21.15.170:6801/2433496486","active_change":"2023-10-20T03:34:52.688943+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4101,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":1989168913}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":681257501}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":8745132}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.170:0","nonce":2182743335}]}]} 2023-10-20T03:35:05.491 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-20T03:35:05.492 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-20T03:35:05.492 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:35:05.791 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:35:05.791 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":27,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:35:05.083690+0000","last_up_change":"2023-10-20T03:34:59.991660+0000","last_in_change":"2023-10-20T03:34:52.643595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":5.9499998092651367,"score_stable":5.9499998092651367,"optimal_score":0.62999999523162842,"raw_score_acting":3.75,"raw_score_stable":3.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2603014448},{"type":"v1","addr":"172.21.15.64:6809","nonce":2603014448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2603014448},{"type":"v1","addr":"172.21.15.64:6811","nonce":2603014448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2603014448},{"type":"v1","addr":"172.21.15.64:6815","nonce":2603014448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2603014448},{"type":"v1","addr":"172.21.15.64:6813","nonce":2603014448}]},"public_addr":"172.21.15.64:6809/2603014448","cluster_addr":"172.21.15.64:6811/2603014448","heartbeat_back_addr":"172.21.15.64:6815/2603014448","heartbeat_front_addr":"172.21.15.64:6813/2603014448","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists","up"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.204996+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.164818+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.304156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.175698+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.169661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:57.863151+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.279628+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.238525+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.074220+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.020311+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.177070+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.063374+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.069832+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.130292+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.265757+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.317166+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:35:05.810 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-20T03:35:05.811 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:35:06.113 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:35:06.113 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":28,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:35:06.095344+0000","last_up_change":"2023-10-20T03:34:59.991660+0000","last_in_change":"2023-10-20T03:34:52.643595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":5.9499998092651367,"score_stable":5.9499998092651367,"optimal_score":0.62999999523162842,"raw_score_acting":3.75,"raw_score_stable":3.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2603014448},{"type":"v1","addr":"172.21.15.64:6809","nonce":2603014448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2603014448},{"type":"v1","addr":"172.21.15.64:6811","nonce":2603014448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2603014448},{"type":"v1","addr":"172.21.15.64:6815","nonce":2603014448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2603014448},{"type":"v1","addr":"172.21.15.64:6813","nonce":2603014448}]},"public_addr":"172.21.15.64:6809/2603014448","cluster_addr":"172.21.15.64:6811/2603014448","heartbeat_back_addr":"172.21.15.64:6815/2603014448","heartbeat_front_addr":"172.21.15.64:6813/2603014448","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists","up"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.204996+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.164818+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.304156+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.175698+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.169661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:57.863151+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.279628+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.238525+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.074220+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.020311+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.177070+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.063374+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.069832+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.130292+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.265757+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:34:58.317166+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:35:06.132 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-20T03:35:06.133 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-20T03:35:06.133 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-20T03:35:06.134 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-20T03:35:06.135 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-20T03:35:06.135 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-20T03:35:06.136 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-20T03:35:06.136 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-20T03:35:06.137 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2023-10-20T03:35:06.137 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2023-10-20T03:35:06.138 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2023-10-20T03:35:06.139 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2023-10-20T03:35:06.139 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2023-10-20T03:35:06.140 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2023-10-20T03:35:06.140 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2023-10-20T03:35:06.141 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2023-10-20T03:35:06.509 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2023-10-20T03:35:06.510 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-20T03:35:06.573 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:06.574 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-20T03:35:06.597 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:06.597 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-20T03:35:06.604 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:06.604 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-20T03:35:06.616 INFO:teuthology.orchestra.run.smithi033.stdout:94489280517 2023-10-20T03:35:06.616 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2023-10-20T03:35:06.643 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2023-10-20T03:35:06.644 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-20T03:35:06.649 INFO:teuthology.orchestra.run.smithi033.stdout:94489280517 2023-10-20T03:35:06.649 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2023-10-20T03:35:06.678 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:06.678 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-20T03:35:06.690 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:06.691 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-20T03:35:06.696 INFO:teuthology.orchestra.run.smithi033.stdout:90194313221 2023-10-20T03:35:06.696 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-20T03:35:06.702 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:06.703 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2023-10-20T03:35:06.710 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:06.711 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2023-10-20T03:35:06.725 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2023-10-20T03:35:06.725 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2023-10-20T03:35:06.733 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2023-10-20T03:35:06.734 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2023-10-20T03:35:06.742 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:06.743 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2023-10-20T03:35:06.763 INFO:teuthology.orchestra.run.smithi033.stdout:94489280517 2023-10-20T03:35:06.763 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2023-10-20T03:35:07.352 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:07.377 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:07.391 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.2 2023-10-20T03:35:07.391 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.453 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.4 2023-10-20T03:35:07.454 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.525 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2023-10-20T03:35:07.542 INFO:teuthology.orchestra.run.smithi033.stdout:94489280517 2023-10-20T03:35:07.600 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.11 2023-10-20T03:35:07.601 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.601 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.6 2023-10-20T03:35:07.602 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.611 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:07.649 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2023-10-20T03:35:07.661 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.3 2023-10-20T03:35:07.662 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.684 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:07.686 INFO:teuthology.orchestra.run.smithi033.stdout:94489280517 2023-10-20T03:35:07.693 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.7 2023-10-20T03:35:07.694 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.706 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:07.713 INFO:teuthology.orchestra.run.smithi033.stdout:90194313221 2023-10-20T03:35:07.714 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2023-10-20T03:35:07.740 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.1 2023-10-20T03:35:07.740 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.745 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.14 2023-10-20T03:35:07.745 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.749 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:07.751 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.12 2023-10-20T03:35:07.752 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.752 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313221 for osd.5 2023-10-20T03:35:07.753 INFO:teuthology.orchestra.run.smithi033.stdout:94489280515 2023-10-20T03:35:07.753 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.757 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.9 2023-10-20T03:35:07.757 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.765 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:07.771 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.0 2023-10-20T03:35:07.771 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.783 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.10 2023-10-20T03:35:07.783 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.784 INFO:teuthology.orchestra.run.smithi033.stdout:94489280517 2023-10-20T03:35:07.785 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.13 2023-10-20T03:35:07.785 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.799 INFO:teuthology.orchestra.run.smithi033.stdout:94489280516 2023-10-20T03:35:07.800 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.15 2023-10-20T03:35:07.800 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.816 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.8 2023-10-20T03:35:07.816 DEBUG:teuthology.parallel:result is None 2023-10-20T03:35:07.816 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-20T03:35:07.817 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T03:35:08.142 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:35:08.143 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-10-20T03:35:08.158 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-20T03:35:06.701085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":436616,"kb_used_data":4144,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512824,"statfs":{"total":1546188226560,"available":1545741131776,"internally_reserved":0,"allocated":4243456,"data_stored":2815068,"data_compressed":165370,"data_compressed_allocated":397312,"data_compressed_original":1687552,"omap_allocated":25502,"internal_metadata":442473570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.590979"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.201176+0000","last_change":"2023-10-20T03:35:05.199365+0000","last_active":"2023-10-20T03:35:05.201176+0000","last_peered":"2023-10-20T03:35:05.201176+0000","last_clean":"2023-10-20T03:35:05.201176+0000","last_became_active":"2023-10-20T03:35:05.198615+0000","last_became_peered":"2023-10-20T03:35:05.198615+0000","last_unstale":"2023-10-20T03:35:05.201176+0000","last_undegraded":"2023-10-20T03:35:05.201176+0000","last_fullsized":"2023-10-20T03:35:05.201176+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:16.347472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.205287+0000","last_change":"2023-10-20T03:35:05.204140+0000","last_active":"2023-10-20T03:35:05.205287+0000","last_peered":"2023-10-20T03:35:05.205287+0000","last_clean":"2023-10-20T03:35:05.205287+0000","last_became_active":"2023-10-20T03:35:05.203380+0000","last_became_peered":"2023-10-20T03:35:05.203380+0000","last_unstale":"2023-10-20T03:35:05.205287+0000","last_undegraded":"2023-10-20T03:35:05.205287+0000","last_fullsized":"2023-10-20T03:35:05.205287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:25.700389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.123744+0000","last_change":"2023-10-20T03:35:05.122638+0000","last_active":"2023-10-20T03:35:05.123744+0000","last_peered":"2023-10-20T03:35:05.123744+0000","last_clean":"2023-10-20T03:35:05.123744+0000","last_became_active":"2023-10-20T03:35:05.122056+0000","last_became_peered":"2023-10-20T03:35:05.122056+0000","last_unstale":"2023-10-20T03:35:05.123744+0000","last_undegraded":"2023-10-20T03:35:05.123744+0000","last_fullsized":"2023-10-20T03:35:05.123744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:18.280647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.116629+0000","last_change":"2023-10-20T03:35:05.114312+0000","last_active":"2023-10-20T03:35:05.116629+0000","last_peered":"2023-10-20T03:35:05.116629+0000","last_clean":"2023-10-20T03:35:05.116629+0000","last_became_active":"2023-10-20T03:35:05.113456+0000","last_became_peered":"2023-10-20T03:35:05.113456+0000","last_unstale":"2023-10-20T03:35:05.116629+0000","last_undegraded":"2023-10-20T03:35:05.116629+0000","last_fullsized":"2023-10-20T03:35:05.116629+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:18.855536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.116107+0000","last_change":"2023-10-20T03:35:05.114672+0000","last_active":"2023-10-20T03:35:05.116107+0000","last_peered":"2023-10-20T03:35:05.116107+0000","last_clean":"2023-10-20T03:35:05.116107+0000","last_became_active":"2023-10-20T03:35:05.114006+0000","last_became_peered":"2023-10-20T03:35:05.114006+0000","last_unstale":"2023-10-20T03:35:05.116107+0000","last_undegraded":"2023-10-20T03:35:05.116107+0000","last_fullsized":"2023-10-20T03:35:05.116107+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:33.806788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-20T03:35:06.477153+0000","last_change":"2023-10-20T03:35:05.113211+0000","last_active":"2023-10-20T03:35:06.477153+0000","last_peered":"2023-10-20T03:35:06.477153+0000","last_clean":"2023-10-20T03:35:06.477153+0000","last_became_active":"2023-10-20T03:35:05.112592+0000","last_became_peered":"2023-10-20T03:35:05.112592+0000","last_unstale":"2023-10-20T03:35:06.477153+0000","last_undegraded":"2023-10-20T03:35:06.477153+0000","last_fullsized":"2023-10-20T03:35:06.477153+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:27.383114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.202299+0000","last_change":"2023-10-20T03:35:05.201208+0000","last_active":"2023-10-20T03:35:05.202299+0000","last_peered":"2023-10-20T03:35:05.202299+0000","last_clean":"2023-10-20T03:35:05.202299+0000","last_became_active":"2023-10-20T03:35:05.200512+0000","last_became_peered":"2023-10-20T03:35:05.200512+0000","last_unstale":"2023-10-20T03:35:05.202299+0000","last_undegraded":"2023-10-20T03:35:05.202299+0000","last_fullsized":"2023-10-20T03:35:05.202299+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:15.820543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-20T03:35:06.477236+0000","last_change":"2023-10-20T03:35:05.117915+0000","last_active":"2023-10-20T03:35:06.477236+0000","last_peered":"2023-10-20T03:35:06.477236+0000","last_clean":"2023-10-20T03:35:06.477236+0000","last_became_active":"2023-10-20T03:35:05.117258+0000","last_became_peered":"2023-10-20T03:35:05.117258+0000","last_unstale":"2023-10-20T03:35:06.477236+0000","last_undegraded":"2023-10-20T03:35:06.477236+0000","last_fullsized":"2023-10-20T03:35:06.477236+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:32.359335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":210,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.993010+0000","last_change":"2023-10-20T03:35:02.275055+0000","last_active":"2023-10-20T03:35:05.993010+0000","last_peered":"2023-10-20T03:35:05.993010+0000","last_clean":"2023-10-20T03:35:05.993010+0000","last_became_active":"2023-10-20T03:35:02.272812+0000","last_became_peered":"2023-10-20T03:35:02.272812+0000","last_unstale":"2023-10-20T03:35:05.993010+0000","last_undegraded":"2023-10-20T03:35:05.993010+0000","last_fullsized":"2023-10-20T03:35:05.993010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_clean_scrub_stamp":"2023-10-20T03:35:01.005741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:02.706360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15078,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.417}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.835}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.742}]}]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":311296,"data_stored":638199,"data_compressed":16815,"data_compressed_allocated":53248,"data_compressed_original":552960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,6,7,8,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.77}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.321}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8979999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5049999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6359999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6699999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5509999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8400000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.889}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]}]},{"osd":5,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":108685,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0339999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.583}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8300000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.766}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5699999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2789999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.331}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.357}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.573}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9179999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8260000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.948}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]}]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.383}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9470000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.857}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3679999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.573}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4129999999999998}]}]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.466}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]}]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]}]},{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":323584,"data_stored":647661,"data_compressed":18691,"data_compressed_allocated":57344,"data_compressed_original":561152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,3,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4139999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7290000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7989999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.738}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9089999999999998}]}]},{"osd":7,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.736}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5960000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]}]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.556}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.859}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.532}]}]},{"osd":11,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.639}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.744}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7949999999999999}]}]},{"osd":12,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,7,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T03:35:08.161 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T03:35:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:35:08.449 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-10-20T03:35:08.466 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-20T03:35:06.701085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":436616,"kb_used_data":4144,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512824,"statfs":{"total":1546188226560,"available":1545741131776,"internally_reserved":0,"allocated":4243456,"data_stored":2815068,"data_compressed":165370,"data_compressed_allocated":397312,"data_compressed_original":1687552,"omap_allocated":25502,"internal_metadata":442473570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.590979"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.201176+0000","last_change":"2023-10-20T03:35:05.199365+0000","last_active":"2023-10-20T03:35:05.201176+0000","last_peered":"2023-10-20T03:35:05.201176+0000","last_clean":"2023-10-20T03:35:05.201176+0000","last_became_active":"2023-10-20T03:35:05.198615+0000","last_became_peered":"2023-10-20T03:35:05.198615+0000","last_unstale":"2023-10-20T03:35:05.201176+0000","last_undegraded":"2023-10-20T03:35:05.201176+0000","last_fullsized":"2023-10-20T03:35:05.201176+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:16.347472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.205287+0000","last_change":"2023-10-20T03:35:05.204140+0000","last_active":"2023-10-20T03:35:05.205287+0000","last_peered":"2023-10-20T03:35:05.205287+0000","last_clean":"2023-10-20T03:35:05.205287+0000","last_became_active":"2023-10-20T03:35:05.203380+0000","last_became_peered":"2023-10-20T03:35:05.203380+0000","last_unstale":"2023-10-20T03:35:05.205287+0000","last_undegraded":"2023-10-20T03:35:05.205287+0000","last_fullsized":"2023-10-20T03:35:05.205287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:25.700389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.123744+0000","last_change":"2023-10-20T03:35:05.122638+0000","last_active":"2023-10-20T03:35:05.123744+0000","last_peered":"2023-10-20T03:35:05.123744+0000","last_clean":"2023-10-20T03:35:05.123744+0000","last_became_active":"2023-10-20T03:35:05.122056+0000","last_became_peered":"2023-10-20T03:35:05.122056+0000","last_unstale":"2023-10-20T03:35:05.123744+0000","last_undegraded":"2023-10-20T03:35:05.123744+0000","last_fullsized":"2023-10-20T03:35:05.123744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:18.280647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.116629+0000","last_change":"2023-10-20T03:35:05.114312+0000","last_active":"2023-10-20T03:35:05.116629+0000","last_peered":"2023-10-20T03:35:05.116629+0000","last_clean":"2023-10-20T03:35:05.116629+0000","last_became_active":"2023-10-20T03:35:05.113456+0000","last_became_peered":"2023-10-20T03:35:05.113456+0000","last_unstale":"2023-10-20T03:35:05.116629+0000","last_undegraded":"2023-10-20T03:35:05.116629+0000","last_fullsized":"2023-10-20T03:35:05.116629+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:18.855536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.116107+0000","last_change":"2023-10-20T03:35:05.114672+0000","last_active":"2023-10-20T03:35:05.116107+0000","last_peered":"2023-10-20T03:35:05.116107+0000","last_clean":"2023-10-20T03:35:05.116107+0000","last_became_active":"2023-10-20T03:35:05.114006+0000","last_became_peered":"2023-10-20T03:35:05.114006+0000","last_unstale":"2023-10-20T03:35:05.116107+0000","last_undegraded":"2023-10-20T03:35:05.116107+0000","last_fullsized":"2023-10-20T03:35:05.116107+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:33.806788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-20T03:35:06.477153+0000","last_change":"2023-10-20T03:35:05.113211+0000","last_active":"2023-10-20T03:35:06.477153+0000","last_peered":"2023-10-20T03:35:06.477153+0000","last_clean":"2023-10-20T03:35:06.477153+0000","last_became_active":"2023-10-20T03:35:05.112592+0000","last_became_peered":"2023-10-20T03:35:05.112592+0000","last_unstale":"2023-10-20T03:35:06.477153+0000","last_undegraded":"2023-10-20T03:35:06.477153+0000","last_fullsized":"2023-10-20T03:35:06.477153+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:27.383114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.202299+0000","last_change":"2023-10-20T03:35:05.201208+0000","last_active":"2023-10-20T03:35:05.202299+0000","last_peered":"2023-10-20T03:35:05.202299+0000","last_clean":"2023-10-20T03:35:05.202299+0000","last_became_active":"2023-10-20T03:35:05.200512+0000","last_became_peered":"2023-10-20T03:35:05.200512+0000","last_unstale":"2023-10-20T03:35:05.202299+0000","last_undegraded":"2023-10-20T03:35:05.202299+0000","last_fullsized":"2023-10-20T03:35:05.202299+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:15.820543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-20T03:35:06.477236+0000","last_change":"2023-10-20T03:35:05.117915+0000","last_active":"2023-10-20T03:35:06.477236+0000","last_peered":"2023-10-20T03:35:06.477236+0000","last_clean":"2023-10-20T03:35:06.477236+0000","last_became_active":"2023-10-20T03:35:05.117258+0000","last_became_peered":"2023-10-20T03:35:05.117258+0000","last_unstale":"2023-10-20T03:35:06.477236+0000","last_undegraded":"2023-10-20T03:35:06.477236+0000","last_fullsized":"2023-10-20T03:35:06.477236+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:32.359335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":210,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.993010+0000","last_change":"2023-10-20T03:35:02.275055+0000","last_active":"2023-10-20T03:35:05.993010+0000","last_peered":"2023-10-20T03:35:05.993010+0000","last_clean":"2023-10-20T03:35:05.993010+0000","last_became_active":"2023-10-20T03:35:02.272812+0000","last_became_peered":"2023-10-20T03:35:02.272812+0000","last_unstale":"2023-10-20T03:35:05.993010+0000","last_undegraded":"2023-10-20T03:35:05.993010+0000","last_fullsized":"2023-10-20T03:35:05.993010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_clean_scrub_stamp":"2023-10-20T03:35:01.005741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:02.706360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15078,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.417}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.835}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.742}]}]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":311296,"data_stored":638199,"data_compressed":16815,"data_compressed_allocated":53248,"data_compressed_original":552960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,6,7,8,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.77}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.321}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8979999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5049999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6359999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6699999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5509999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8400000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.889}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]}]},{"osd":5,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":108685,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0339999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.583}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8300000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.766}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5699999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2789999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.331}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.357}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.573}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9179999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8260000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.948}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]}]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.383}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9470000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.857}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3679999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.573}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4129999999999998}]}]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.466}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]}]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]}]},{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":323584,"data_stored":647661,"data_compressed":18691,"data_compressed_allocated":57344,"data_compressed_original":561152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,3,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4139999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7290000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7989999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.738}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9089999999999998}]}]},{"osd":7,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.736}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5960000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]}]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.556}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.859}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.532}]}]},{"osd":11,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.639}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.744}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7949999999999999}]}]},{"osd":12,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,7,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T03:35:08.468 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-20T03:35:08.468 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-20T03:35:08.468 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-20T03:35:08.469 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-20T03:35:08.783 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:35:08.783 INFO:teuthology.orchestra.run.smithi033.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-20T03:35:08.800 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-20T03:35:08.801 INFO:teuthology.run_tasks:Running task thrashosds... 2023-10-20T03:35:08.812 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2023-10-20T03:35:08.813 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri10-testing-2023-10-11-0812'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '8a5df55d22d3638066184c241442ff00fdc0ebdd'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8a5df55d22d3638066184c241442ff00fdc0ebdd'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri10-testing-2023-10-11-0812', 'sha1': '8a5df55d22d3638066184c241442ff00fdc0ebdd'}} 2023-10-20T03:35:08.813 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2023-10-20T03:35:08.813 INFO:tasks.thrashosds:Beginning thrashosds... 2023-10-20T03:35:08.814 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-20T03:35:08.814 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T03:35:09.099 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:35:09.099 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-10-20T03:35:09.116 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-10-20T03:35:08.701813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":436616,"kb_used_data":4144,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512824,"statfs":{"total":1546188226560,"available":1545741131776,"internally_reserved":0,"allocated":4243456,"data_stored":2815068,"data_compressed":165370,"data_compressed_allocated":397312,"data_compressed_original":1687552,"omap_allocated":25502,"internal_metadata":442473570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.591707"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.201176+0000","last_change":"2023-10-20T03:35:05.199365+0000","last_active":"2023-10-20T03:35:05.201176+0000","last_peered":"2023-10-20T03:35:05.201176+0000","last_clean":"2023-10-20T03:35:05.201176+0000","last_became_active":"2023-10-20T03:35:05.198615+0000","last_became_peered":"2023-10-20T03:35:05.198615+0000","last_unstale":"2023-10-20T03:35:05.201176+0000","last_undegraded":"2023-10-20T03:35:05.201176+0000","last_fullsized":"2023-10-20T03:35:05.201176+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:16.347472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.205287+0000","last_change":"2023-10-20T03:35:05.204140+0000","last_active":"2023-10-20T03:35:05.205287+0000","last_peered":"2023-10-20T03:35:05.205287+0000","last_clean":"2023-10-20T03:35:05.205287+0000","last_became_active":"2023-10-20T03:35:05.203380+0000","last_became_peered":"2023-10-20T03:35:05.203380+0000","last_unstale":"2023-10-20T03:35:05.205287+0000","last_undegraded":"2023-10-20T03:35:05.205287+0000","last_fullsized":"2023-10-20T03:35:05.205287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:25.700389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.123744+0000","last_change":"2023-10-20T03:35:05.122638+0000","last_active":"2023-10-20T03:35:05.123744+0000","last_peered":"2023-10-20T03:35:05.123744+0000","last_clean":"2023-10-20T03:35:05.123744+0000","last_became_active":"2023-10-20T03:35:05.122056+0000","last_became_peered":"2023-10-20T03:35:05.122056+0000","last_unstale":"2023-10-20T03:35:05.123744+0000","last_undegraded":"2023-10-20T03:35:05.123744+0000","last_fullsized":"2023-10-20T03:35:05.123744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:18.280647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.116629+0000","last_change":"2023-10-20T03:35:05.114312+0000","last_active":"2023-10-20T03:35:05.116629+0000","last_peered":"2023-10-20T03:35:05.116629+0000","last_clean":"2023-10-20T03:35:05.116629+0000","last_became_active":"2023-10-20T03:35:05.113456+0000","last_became_peered":"2023-10-20T03:35:05.113456+0000","last_unstale":"2023-10-20T03:35:05.116629+0000","last_undegraded":"2023-10-20T03:35:05.116629+0000","last_fullsized":"2023-10-20T03:35:05.116629+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:18.855536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.116107+0000","last_change":"2023-10-20T03:35:05.114672+0000","last_active":"2023-10-20T03:35:05.116107+0000","last_peered":"2023-10-20T03:35:05.116107+0000","last_clean":"2023-10-20T03:35:05.116107+0000","last_became_active":"2023-10-20T03:35:05.114006+0000","last_became_peered":"2023-10-20T03:35:05.114006+0000","last_unstale":"2023-10-20T03:35:05.116107+0000","last_undegraded":"2023-10-20T03:35:05.116107+0000","last_fullsized":"2023-10-20T03:35:05.116107+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:33.806788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-20T03:35:06.477153+0000","last_change":"2023-10-20T03:35:05.113211+0000","last_active":"2023-10-20T03:35:06.477153+0000","last_peered":"2023-10-20T03:35:06.477153+0000","last_clean":"2023-10-20T03:35:06.477153+0000","last_became_active":"2023-10-20T03:35:05.112592+0000","last_became_peered":"2023-10-20T03:35:05.112592+0000","last_unstale":"2023-10-20T03:35:06.477153+0000","last_undegraded":"2023-10-20T03:35:06.477153+0000","last_fullsized":"2023-10-20T03:35:06.477153+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:27.383114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.202299+0000","last_change":"2023-10-20T03:35:05.201208+0000","last_active":"2023-10-20T03:35:05.202299+0000","last_peered":"2023-10-20T03:35:05.202299+0000","last_clean":"2023-10-20T03:35:05.202299+0000","last_became_active":"2023-10-20T03:35:05.200512+0000","last_became_peered":"2023-10-20T03:35:05.200512+0000","last_unstale":"2023-10-20T03:35:05.202299+0000","last_undegraded":"2023-10-20T03:35:05.202299+0000","last_fullsized":"2023-10-20T03:35:05.202299+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:15.820543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-20T03:35:06.477236+0000","last_change":"2023-10-20T03:35:05.117915+0000","last_active":"2023-10-20T03:35:06.477236+0000","last_peered":"2023-10-20T03:35:06.477236+0000","last_clean":"2023-10-20T03:35:06.477236+0000","last_became_active":"2023-10-20T03:35:05.117258+0000","last_became_peered":"2023-10-20T03:35:05.117258+0000","last_unstale":"2023-10-20T03:35:06.477236+0000","last_undegraded":"2023-10-20T03:35:06.477236+0000","last_fullsized":"2023-10-20T03:35:06.477236+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:32.359335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":210,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.993010+0000","last_change":"2023-10-20T03:35:02.275055+0000","last_active":"2023-10-20T03:35:05.993010+0000","last_peered":"2023-10-20T03:35:05.993010+0000","last_clean":"2023-10-20T03:35:05.993010+0000","last_became_active":"2023-10-20T03:35:02.272812+0000","last_became_peered":"2023-10-20T03:35:02.272812+0000","last_unstale":"2023-10-20T03:35:05.993010+0000","last_undegraded":"2023-10-20T03:35:05.993010+0000","last_fullsized":"2023-10-20T03:35:05.993010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_clean_scrub_stamp":"2023-10-20T03:35:01.005741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:02.706360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15078,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.417}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.835}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.742}]}]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":311296,"data_stored":638199,"data_compressed":16815,"data_compressed_allocated":53248,"data_compressed_original":552960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,6,7,8,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.77}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.321}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8979999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5049999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6359999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6699999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5509999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8400000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.889}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]}]},{"osd":5,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":108685,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0339999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.583}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8300000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.766}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5699999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2789999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.331}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.357}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.573}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9179999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8260000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.948}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]}]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.383}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9470000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.857}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3679999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.573}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4129999999999998}]}]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.466}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]}]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]}]},{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":323584,"data_stored":647661,"data_compressed":18691,"data_compressed_allocated":57344,"data_compressed_original":561152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,3,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4139999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7290000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7989999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.738}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9089999999999998}]}]},{"osd":7,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.736}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5960000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]}]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.556}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.859}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.532}]}]},{"osd":11,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.639}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.744}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7949999999999999}]}]},{"osd":12,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,7,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T03:35:09.119 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T03:35:09.411 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:35:09.412 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-10-20T03:35:09.428 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-10-20T03:35:08.701813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":436616,"kb_used_data":4144,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509512824,"statfs":{"total":1546188226560,"available":1545741131776,"internally_reserved":0,"allocated":4243456,"data_stored":2815068,"data_compressed":165370,"data_compressed_allocated":397312,"data_compressed_original":1687552,"omap_allocated":25502,"internal_metadata":442473570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.591707"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.201176+0000","last_change":"2023-10-20T03:35:05.199365+0000","last_active":"2023-10-20T03:35:05.201176+0000","last_peered":"2023-10-20T03:35:05.201176+0000","last_clean":"2023-10-20T03:35:05.201176+0000","last_became_active":"2023-10-20T03:35:05.198615+0000","last_became_peered":"2023-10-20T03:35:05.198615+0000","last_unstale":"2023-10-20T03:35:05.201176+0000","last_undegraded":"2023-10-20T03:35:05.201176+0000","last_fullsized":"2023-10-20T03:35:05.201176+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:16.347472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.205287+0000","last_change":"2023-10-20T03:35:05.204140+0000","last_active":"2023-10-20T03:35:05.205287+0000","last_peered":"2023-10-20T03:35:05.205287+0000","last_clean":"2023-10-20T03:35:05.205287+0000","last_became_active":"2023-10-20T03:35:05.203380+0000","last_became_peered":"2023-10-20T03:35:05.203380+0000","last_unstale":"2023-10-20T03:35:05.205287+0000","last_undegraded":"2023-10-20T03:35:05.205287+0000","last_fullsized":"2023-10-20T03:35:05.205287+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:25.700389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.123744+0000","last_change":"2023-10-20T03:35:05.122638+0000","last_active":"2023-10-20T03:35:05.123744+0000","last_peered":"2023-10-20T03:35:05.123744+0000","last_clean":"2023-10-20T03:35:05.123744+0000","last_became_active":"2023-10-20T03:35:05.122056+0000","last_became_peered":"2023-10-20T03:35:05.122056+0000","last_unstale":"2023-10-20T03:35:05.123744+0000","last_undegraded":"2023-10-20T03:35:05.123744+0000","last_fullsized":"2023-10-20T03:35:05.123744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:18.280647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.116629+0000","last_change":"2023-10-20T03:35:05.114312+0000","last_active":"2023-10-20T03:35:05.116629+0000","last_peered":"2023-10-20T03:35:05.116629+0000","last_clean":"2023-10-20T03:35:05.116629+0000","last_became_active":"2023-10-20T03:35:05.113456+0000","last_became_peered":"2023-10-20T03:35:05.113456+0000","last_unstale":"2023-10-20T03:35:05.116629+0000","last_undegraded":"2023-10-20T03:35:05.116629+0000","last_fullsized":"2023-10-20T03:35:05.116629+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:18.855536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.116107+0000","last_change":"2023-10-20T03:35:05.114672+0000","last_active":"2023-10-20T03:35:05.116107+0000","last_peered":"2023-10-20T03:35:05.116107+0000","last_clean":"2023-10-20T03:35:05.116107+0000","last_became_active":"2023-10-20T03:35:05.114006+0000","last_became_peered":"2023-10-20T03:35:05.114006+0000","last_unstale":"2023-10-20T03:35:05.116107+0000","last_undegraded":"2023-10-20T03:35:05.116107+0000","last_fullsized":"2023-10-20T03:35:05.116107+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:33.806788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-20T03:35:06.477153+0000","last_change":"2023-10-20T03:35:05.113211+0000","last_active":"2023-10-20T03:35:06.477153+0000","last_peered":"2023-10-20T03:35:06.477153+0000","last_clean":"2023-10-20T03:35:06.477153+0000","last_became_active":"2023-10-20T03:35:05.112592+0000","last_became_peered":"2023-10-20T03:35:05.112592+0000","last_unstale":"2023-10-20T03:35:06.477153+0000","last_undegraded":"2023-10-20T03:35:06.477153+0000","last_fullsized":"2023-10-20T03:35:06.477153+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:27.383114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.202299+0000","last_change":"2023-10-20T03:35:05.201208+0000","last_active":"2023-10-20T03:35:05.202299+0000","last_peered":"2023-10-20T03:35:05.202299+0000","last_clean":"2023-10-20T03:35:05.202299+0000","last_became_active":"2023-10-20T03:35:05.200512+0000","last_became_peered":"2023-10-20T03:35:05.200512+0000","last_unstale":"2023-10-20T03:35:05.202299+0000","last_undegraded":"2023-10-20T03:35:05.202299+0000","last_fullsized":"2023-10-20T03:35:05.202299+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:15.820543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-20T03:35:06.477236+0000","last_change":"2023-10-20T03:35:05.117915+0000","last_active":"2023-10-20T03:35:06.477236+0000","last_peered":"2023-10-20T03:35:06.477236+0000","last_clean":"2023-10-20T03:35:06.477236+0000","last_became_active":"2023-10-20T03:35:05.117258+0000","last_became_peered":"2023-10-20T03:35:05.117258+0000","last_unstale":"2023-10-20T03:35:06.477236+0000","last_undegraded":"2023-10-20T03:35:06.477236+0000","last_fullsized":"2023-10-20T03:35:06.477236+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:32.359335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":210,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-20T03:35:05.993010+0000","last_change":"2023-10-20T03:35:02.275055+0000","last_active":"2023-10-20T03:35:05.993010+0000","last_peered":"2023-10-20T03:35:05.993010+0000","last_clean":"2023-10-20T03:35:05.993010+0000","last_became_active":"2023-10-20T03:35:02.272812+0000","last_became_peered":"2023-10-20T03:35:02.272812+0000","last_unstale":"2023-10-20T03:35:05.993010+0000","last_undegraded":"2023-10-20T03:35:05.993010+0000","last_fullsized":"2023-10-20T03:35:05.993010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_clean_scrub_stamp":"2023-10-20T03:35:01.005741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:02.706360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15078,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.417}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.835}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.742}]}]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":311296,"data_stored":638199,"data_compressed":16815,"data_compressed_allocated":53248,"data_compressed_original":552960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,6,7,8,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.77}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.321}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8979999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5049999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6359999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6699999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5509999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8400000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.889}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.796}]}]},{"osd":5,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":108685,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0339999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.583}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9019999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8300000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.766}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3250000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5699999999999998}]}]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2789999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.331}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.357}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.573}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7150000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9179999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8260000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.948}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]}]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5269999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.383}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9470000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.857}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3679999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.573}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4129999999999998}]}]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.466}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]}]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]}]},{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":323584,"data_stored":647661,"data_compressed":18691,"data_compressed_allocated":57344,"data_compressed_original":561152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,3,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4139999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7290000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7989999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.738}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9089999999999998}]}]},{"osd":7,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,1,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.736}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3480000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5960000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]}]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,7,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.556}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3700000000000001}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.629}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6919999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.859}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.532}]}]},{"osd":11,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.639}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.744}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7949999999999999}]}]},{"osd":12,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":109271,"data_compressed":9276,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,7,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T03:35:09.430 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-20T03:35:09.431 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2023-10-20T03:35:09.717 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 28 2023-10-20T03:35:09.718 INFO:teuthology.orchestra.run.smithi033.stdout:fsid b920f165-e9e7-4c3e-8d7b-cfdc002ae589 2023-10-20T03:35:09.718 INFO:teuthology.orchestra.run.smithi033.stdout:created 2023-10-20T03:34:47.883622+0000 2023-10-20T03:35:09.718 INFO:teuthology.orchestra.run.smithi033.stdout:modified 2023-10-20T03:35:06.095344+0000 2023-10-20T03:35:09.718 INFO:teuthology.orchestra.run.smithi033.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-10-20T03:35:09.719 INFO:teuthology.orchestra.run.smithi033.stdout:crush_version 5 2023-10-20T03:35:09.719 INFO:teuthology.orchestra.run.smithi033.stdout:full_ratio 0.95 2023-10-20T03:35:09.719 INFO:teuthology.orchestra.run.smithi033.stdout:backfillfull_ratio 0.9 2023-10-20T03:35:09.719 INFO:teuthology.orchestra.run.smithi033.stdout:nearfull_ratio 0.85 2023-10-20T03:35:09.719 INFO:teuthology.orchestra.run.smithi033.stdout:require_min_compat_client luminous 2023-10-20T03:35:09.720 INFO:teuthology.orchestra.run.smithi033.stdout:min_compat_client jewel 2023-10-20T03:35:09.720 INFO:teuthology.orchestra.run.smithi033.stdout:require_osd_release reef 2023-10-20T03:35:09.720 INFO:teuthology.orchestra.run.smithi033.stdout:stretch_mode_enabled false 2023-10-20T03:35:09.720 INFO:teuthology.orchestra.run.smithi033.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2023-10-20T03:35:09.720 INFO:teuthology.orchestra.run.smithi033.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool stripe_width 0 application rbd read_balance_score 5.95 2023-10-20T03:35:09.720 INFO:teuthology.orchestra.run.smithi033.stdout:max_osd 16 2023-10-20T03:35:09.721 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6824/121544157,v1:172.21.15.33:6825/121544157] [v2:172.21.15.33:6826/121544157,v1:172.21.15.33:6827/121544157] exists,up 4872451c-1bc7-4f21-a95f-986fa428ea65 2023-10-20T03:35:09.721 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6808/562043184,v1:172.21.15.40:6809/562043184] [v2:172.21.15.40:6810/562043184,v1:172.21.15.40:6811/562043184] exists,up 0b570426-e165-4caa-a914-e1bed7d805fc 2023-10-20T03:35:09.721 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6824/3785405088,v1:172.21.15.64:6825/3785405088] [v2:172.21.15.64:6826/3785405088,v1:172.21.15.64:6827/3785405088] exists,up e0184c10-844b-4bcb-adaf-a0d407b708ee 2023-10-20T03:35:09.721 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6818/1426674535,v1:172.21.15.170:6820/1426674535] [v2:172.21.15.170:6821/1426674535,v1:172.21.15.170:6823/1426674535] exists,up 8120023b-1d48-411d-a642-2529a697b5c9 2023-10-20T03:35:09.721 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6808/3186493546,v1:172.21.15.33:6809/3186493546] [v2:172.21.15.33:6810/3186493546,v1:172.21.15.33:6811/3186493546] exists,up 549b3b39-41d5-4d9b-932e-a7b3e8cc0684 2023-10-20T03:35:09.722 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6814/1211670976,v1:172.21.15.40:6817/1211670976] [v2:172.21.15.40:6821/1211670976,v1:172.21.15.40:6823/1211670976] exists,up a99ea3db-6433-4312-83d1-64ba69f028a4 2023-10-20T03:35:09.722 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6 up in weight 1 up_from 22 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6808/2603014448,v1:172.21.15.64:6809/2603014448] [v2:172.21.15.64:6810/2603014448,v1:172.21.15.64:6811/2603014448] exists,up d2b38063-49eb-4f56-bb8d-411419b71b9b 2023-10-20T03:35:09.722 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6810/2804292733,v1:172.21.15.170:6811/2804292733] [v2:172.21.15.170:6812/2804292733,v1:172.21.15.170:6813/2804292733] exists,up 80e34f64-35dc-4a6d-9f7f-c2263c78350d 2023-10-20T03:35:09.722 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6800/1638010589,v1:172.21.15.33:6801/1638010589] [v2:172.21.15.33:6802/1638010589,v1:172.21.15.33:6803/1638010589] exists,up 1cd10625-66a8-4488-b1e5-378cb0062107 2023-10-20T03:35:09.722 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6800/2173797306,v1:172.21.15.40:6801/2173797306] [v2:172.21.15.40:6802/2173797306,v1:172.21.15.40:6803/2173797306] exists,up b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4 2023-10-20T03:35:09.723 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6800/91701057,v1:172.21.15.64:6801/91701057] [v2:172.21.15.64:6802/91701057,v1:172.21.15.64:6803/91701057] exists,up 42a67bcc-5c07-499d-8902-08c2f15e7fbb 2023-10-20T03:35:09.723 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6802/1717984751,v1:172.21.15.170:6803/1717984751] [v2:172.21.15.170:6804/1717984751,v1:172.21.15.170:6805/1717984751] exists,up ab1db647-aada-4deb-abdc-0a1baff548db 2023-10-20T03:35:09.723 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6816/3797427065,v1:172.21.15.33:6817/3797427065] [v2:172.21.15.33:6818/3797427065,v1:172.21.15.33:6819/3797427065] exists,up b20cee74-2cce-4153-97c2-56e9db40c712 2023-10-20T03:35:09.723 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6812/3041424233,v1:172.21.15.40:6816/3041424233] [v2:172.21.15.40:6819/3041424233,v1:172.21.15.40:6822/3041424233] exists,up 0974cd04-c690-4cba-ac05-5d404493dd1d 2023-10-20T03:35:09.723 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6816/2940007286,v1:172.21.15.64:6817/2940007286] [v2:172.21.15.64:6818/2940007286,v1:172.21.15.64:6819/2940007286] exists,up 3bf53124-13b6-4e5c-9868-2382719ffe25 2023-10-20T03:35:09.723 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6819/826332016,v1:172.21.15.170:6822/826332016] [v2:172.21.15.170:6824/826332016,v1:172.21.15.170:6826/826332016] exists,up b0b5ffe5-a25b-412f-818f-b60907c0089b 2023-10-20T03:35:09.735 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6824/121544157,v1:172.21.15.33:6825/121544157] [v2:172.21.15.33:6826/121544157,v1:172.21.15.33:6827/121544157] exists,up 4872451c-1bc7-4f21-a95f-986fa428ea65', 'osd.1 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6808/562043184,v1:172.21.15.40:6809/562043184] [v2:172.21.15.40:6810/562043184,v1:172.21.15.40:6811/562043184] exists,up 0b570426-e165-4caa-a914-e1bed7d805fc', 'osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6824/3785405088,v1:172.21.15.64:6825/3785405088] [v2:172.21.15.64:6826/3785405088,v1:172.21.15.64:6827/3785405088] exists,up e0184c10-844b-4bcb-adaf-a0d407b708ee', 'osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6818/1426674535,v1:172.21.15.170:6820/1426674535] [v2:172.21.15.170:6821/1426674535,v1:172.21.15.170:6823/1426674535] exists,up 8120023b-1d48-411d-a642-2529a697b5c9', 'osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6808/3186493546,v1:172.21.15.33:6809/3186493546] [v2:172.21.15.33:6810/3186493546,v1:172.21.15.33:6811/3186493546] exists,up 549b3b39-41d5-4d9b-932e-a7b3e8cc0684', 'osd.5 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6814/1211670976,v1:172.21.15.40:6817/1211670976] [v2:172.21.15.40:6821/1211670976,v1:172.21.15.40:6823/1211670976] exists,up a99ea3db-6433-4312-83d1-64ba69f028a4', 'osd.6 up in weight 1 up_from 22 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6808/2603014448,v1:172.21.15.64:6809/2603014448] [v2:172.21.15.64:6810/2603014448,v1:172.21.15.64:6811/2603014448] exists,up d2b38063-49eb-4f56-bb8d-411419b71b9b', 'osd.7 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6810/2804292733,v1:172.21.15.170:6811/2804292733] [v2:172.21.15.170:6812/2804292733,v1:172.21.15.170:6813/2804292733] exists,up 80e34f64-35dc-4a6d-9f7f-c2263c78350d', 'osd.8 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6800/1638010589,v1:172.21.15.33:6801/1638010589] [v2:172.21.15.33:6802/1638010589,v1:172.21.15.33:6803/1638010589] exists,up 1cd10625-66a8-4488-b1e5-378cb0062107', 'osd.9 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6800/2173797306,v1:172.21.15.40:6801/2173797306] [v2:172.21.15.40:6802/2173797306,v1:172.21.15.40:6803/2173797306] exists,up b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6800/91701057,v1:172.21.15.64:6801/91701057] [v2:172.21.15.64:6802/91701057,v1:172.21.15.64:6803/91701057] exists,up 42a67bcc-5c07-499d-8902-08c2f15e7fbb', 'osd.11 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6802/1717984751,v1:172.21.15.170:6803/1717984751] [v2:172.21.15.170:6804/1717984751,v1:172.21.15.170:6805/1717984751] exists,up ab1db647-aada-4deb-abdc-0a1baff548db', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6816/3797427065,v1:172.21.15.33:6817/3797427065] [v2:172.21.15.33:6818/3797427065,v1:172.21.15.33:6819/3797427065] exists,up b20cee74-2cce-4153-97c2-56e9db40c712', 'osd.13 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6812/3041424233,v1:172.21.15.40:6816/3041424233] [v2:172.21.15.40:6819/3041424233,v1:172.21.15.40:6822/3041424233] exists,up 0974cd04-c690-4cba-ac05-5d404493dd1d', 'osd.14 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6816/2940007286,v1:172.21.15.64:6817/2940007286] [v2:172.21.15.64:6818/2940007286,v1:172.21.15.64:6819/2940007286] exists,up 3bf53124-13b6-4e5c-9868-2382719ffe25', 'osd.15 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6819/826332016,v1:172.21.15.170:6822/826332016] [v2:172.21.15.170:6824/826332016,v1:172.21.15.170:6826/826332016] exists,up b0b5ffe5-a25b-412f-818f-b60907c0089b'] 2023-10-20T03:35:09.735 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2023-10-20T03:35:09.855 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:09.856 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mon.a", 2023-10-20T03:35:09.856 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster": "ceph", 2023-10-20T03:35:09.856 INFO:teuthology.orchestra.run.smithi033.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2023-10-20T03:35:09.856 INFO:teuthology.orchestra.run.smithi033.stdout: "admin_socket_mode": "", 2023-10-20T03:35:09.856 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2023-10-20T03:35:09.857 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_client_required": "cephx, none", 2023-10-20T03:35:09.857 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_cluster_required": "cephx", 2023-10-20T03:35:09.857 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_debug": "true", 2023-10-20T03:35:09.857 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2023-10-20T03:35:09.857 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_mon_ticket_ttl": "660.000000", 2023-10-20T03:35:09.857 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_service_required": "cephx", 2023-10-20T03:35:09.858 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_service_ticket_ttl": "240.000000", 2023-10-20T03:35:09.858 INFO:teuthology.orchestra.run.smithi033.stdout: "auth_supported": "cephx", 2023-10-20T03:35:09.858 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_aio": "true", 2023-10-20T03:35:09.858 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_aio_max_queue_depth": "1024", 2023-10-20T03:35:09.858 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_aio_poll_ms": "250", 2023-10-20T03:35:09.859 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_aio_reap_max": "16", 2023-10-20T03:35:09.859 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_async_discard": "false", 2023-10-20T03:35:09.859 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_block_size": "4096", 2023-10-20T03:35:09.859 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_debug_aio": "false", 2023-10-20T03:35:09.859 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_debug_aio_log_age": "5.000000", 2023-10-20T03:35:09.860 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2023-10-20T03:35:09.860 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_debug_inflight_ios": "false", 2023-10-20T03:35:09.860 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_enable_discard": "false", 2023-10-20T03:35:09.860 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_flock_retry": "3", 2023-10-20T03:35:09.860 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_flock_retry_interval": "0.100000", 2023-10-20T03:35:09.861 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_inject_crash": "0", 2023-10-20T03:35:09.861 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_inject_crash_flush_delay": "2", 2023-10-20T03:35:09.861 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_ioring": "false", 2023-10-20T03:35:09.861 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_ioring_hipri": "false", 2023-10-20T03:35:09.861 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_ioring_sqthread_poll": "false", 2023-10-20T03:35:09.862 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_nvme_unbind_from_kernel": "false", 2023-10-20T03:35:09.862 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_read_buffer_alignment": "4096", 2023-10-20T03:35:09.862 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_read_preallocated_huge_buffers": "", 2023-10-20T03:35:09.862 INFO:teuthology.orchestra.run.smithi033.stdout: "bdev_type": "", 2023-10-20T03:35:09.862 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_alloc_size": "1048576", 2023-10-20T03:35:09.863 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_allocator": "hybrid", 2023-10-20T03:35:09.863 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_buffered_io": "true", 2023-10-20T03:35:09.863 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_check_for_zeros": "false", 2023-10-20T03:35:09.863 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_check_volume_selector_often": "false", 2023-10-20T03:35:09.863 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_check_volume_selector_on_umount": "false", 2023-10-20T03:35:09.864 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_compact_log_sync": "false", 2023-10-20T03:35:09.864 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2023-10-20T03:35:09.864 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2023-10-20T03:35:09.864 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_log_compact_min_size": "16777216", 2023-10-20T03:35:09.864 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_log_replay_check_allocations": "true", 2023-10-20T03:35:09.864 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_max_log_runway": "4194304", 2023-10-20T03:35:09.865 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_max_prefetch": "1048576", 2023-10-20T03:35:09.865 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_min_flush_size": "524288", 2023-10-20T03:35:09.865 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_min_log_runway": "1048576", 2023-10-20T03:35:09.865 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_replay_recovery": "false", 2023-10-20T03:35:09.865 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_replay_recovery_disable_compact": "false", 2023-10-20T03:35:09.865 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_shared_alloc_size": "65536", 2023-10-20T03:35:09.866 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_sync_write": "false", 2023-10-20T03:35:09.866 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2023-10-20T03:35:09.866 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2023-10-20T03:35:09.866 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2023-10-20T03:35:09.866 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_allocation_from_file": "true", 2023-10-20T03:35:09.867 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_allocator": "hybrid", 2023-10-20T03:35:09.867 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2023-10-20T03:35:09.867 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2023-10-20T03:35:09.867 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2023-10-20T03:35:09.867 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2023-10-20T03:35:09.867 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2023-10-20T03:35:09.868 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bitmapallocator_span_size": "1024", 2023-10-20T03:35:09.868 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blobid_prealloc": "10240", 2023-10-20T03:35:09.868 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_create": "true", 2023-10-20T03:35:09.868 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_db_create": "false", 2023-10-20T03:35:09.868 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_db_path": "", 2023-10-20T03:35:09.868 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_db_size": "0", 2023-10-20T03:35:09.869 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_path": "", 2023-10-20T03:35:09.869 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_preallocate_file": "false", 2023-10-20T03:35:09.869 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_size": "107374182400", 2023-10-20T03:35:09.869 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_wal_create": "false", 2023-10-20T03:35:09.869 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_wal_path": "", 2023-10-20T03:35:09.870 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_block_wal_size": "100663296", 2023-10-20T03:35:09.870 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bluefs": "true", 2023-10-20T03:35:09.870 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2023-10-20T03:35:09.870 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bluefs_env_mirror": "false", 2023-10-20T03:35:09.870 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_bluefs_max_free": "10737418240", 2023-10-20T03:35:09.871 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2023-10-20T03:35:09.871 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2023-10-20T03:35:09.871 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2023-10-20T03:35:09.871 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2023-10-20T03:35:09.871 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2023-10-20T03:35:09.871 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_autotune": "true", 2023-10-20T03:35:09.872 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_autotune_interval": "5.000000", 2023-10-20T03:35:09.872 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2023-10-20T03:35:09.872 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_kv_ratio": "0.450000", 2023-10-20T03:35:09.872 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_ratio": "0.450000", 2023-10-20T03:35:09.872 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_size": "0", 2023-10-20T03:35:09.872 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_size_hdd": "1073741824", 2023-10-20T03:35:09.873 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_size_ssd": "3221225472", 2023-10-20T03:35:09.873 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_trim_interval": "0.050000", 2023-10-20T03:35:09.873 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2023-10-20T03:35:09.873 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_type": "2q", 2023-10-20T03:35:09.873 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2023-10-20T03:35:09.874 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_clone_cow": "true", 2023-10-20T03:35:09.874 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_algorithm": "snappy", 2023-10-20T03:35:09.874 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_max_blob_size": "0", 2023-10-20T03:35:09.874 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2023-10-20T03:35:09.874 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2023-10-20T03:35:09.875 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_min_blob_size": "0", 2023-10-20T03:35:09.875 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2023-10-20T03:35:09.875 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2023-10-20T03:35:09.875 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_mode": "none", 2023-10-20T03:35:09.875 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_compression_required_ratio": "0.875000", 2023-10-20T03:35:09.875 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_csum_type": "crc32c", 2023-10-20T03:35:09.876 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_enforce_settings": "default", 2023-10-20T03:35:09.876 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_freelist": "false", 2023-10-20T03:35:09.876 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_fsck_abort": "false", 2023-10-20T03:35:09.876 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2023-10-20T03:35:09.877 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2023-10-20T03:35:09.877 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_inject_read_err": "false", 2023-10-20T03:35:09.877 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_legacy_omap": "false", 2023-10-20T03:35:09.877 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_no_reuse_blocks": "false", 2023-10-20T03:35:09.877 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_omit_block_device_write": "false", 2023-10-20T03:35:09.878 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_omit_kv_commit": "false", 2023-10-20T03:35:09.878 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2023-10-20T03:35:09.878 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_prefill": "0.000000", 2023-10-20T03:35:09.878 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_prefragment_max": "1048576", 2023-10-20T03:35:09.878 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_random_read_err": "0.000000", 2023-10-20T03:35:09.878 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2023-10-20T03:35:09.879 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_small_allocations": "0", 2023-10-20T03:35:09.879 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2023-10-20T03:35:09.879 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_default_buffered_read": "true", 2023-10-20T03:35:09.879 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_default_buffered_write": "false", 2023-10-20T03:35:09.879 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_deferred_batch_ops": "0", 2023-10-20T03:35:09.880 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2023-10-20T03:35:09.880 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2023-10-20T03:35:09.880 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_elastic_shared_blobs": "true", 2023-10-20T03:35:09.880 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2023-10-20T03:35:09.880 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_shard_max_size": "1200", 2023-10-20T03:35:09.881 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_shard_min_size": "150", 2023-10-20T03:35:09.881 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_shard_target_size": "500", 2023-10-20T03:35:09.881 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2023-10-20T03:35:09.881 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fail_eio": "false", 2023-10-20T03:35:09.881 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_freelist_blocks_per_key": "128", 2023-10-20T03:35:09.882 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2023-10-20T03:35:09.882 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2023-10-20T03:35:09.882 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2023-10-20T03:35:09.882 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_mkfs": "true", 2023-10-20T03:35:09.882 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2023-10-20T03:35:09.883 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_mount": "false", 2023-10-20T03:35:09.883 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_mount_deep": "false", 2023-10-20T03:35:09.883 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_umount": "false", 2023-10-20T03:35:09.883 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_on_umount_deep": "false", 2023-10-20T03:35:09.883 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2023-10-20T03:35:09.884 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_quick_fix_threads": "2", 2023-10-20T03:35:09.884 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2023-10-20T03:35:09.884 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2023-10-20T03:35:09.884 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_gc_enable_blob_threshold": "0", 2023-10-20T03:35:09.884 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_gc_enable_total_threshold": "0", 2023-10-20T03:35:09.885 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2023-10-20T03:35:09.885 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_ignore_data_csum": "false", 2023-10-20T03:35:09.885 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2023-10-20T03:35:09.885 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_kvbackend": "rocksdb", 2023-10-20T03:35:09.885 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_log_collection_list_age": "60.000000", 2023-10-20T03:35:09.886 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2023-10-20T03:35:09.886 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_log_op_age": "5.000000", 2023-10-20T03:35:09.886 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_alloc_size": "0", 2023-10-20T03:35:09.886 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_blob_size": "0", 2023-10-20T03:35:09.886 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_blob_size_hdd": "65536", 2023-10-20T03:35:09.887 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_blob_size_ssd": "65536", 2023-10-20T03:35:09.887 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_defer_interval": "3.000000", 2023-10-20T03:35:09.887 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_max_deferred_txc": "32", 2023-10-20T03:35:09.887 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_min_alloc_size": "0", 2023-10-20T03:35:09.887 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_min_alloc_size_hdd": "4096", 2023-10-20T03:35:09.887 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_min_alloc_size_ssd": "4096", 2023-10-20T03:35:09.888 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_nid_prealloc": "1024", 2023-10-20T03:35:09.888 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_prefer_deferred_size": "0", 2023-10-20T03:35:09.888 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2023-10-20T03:35:09.888 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2023-10-20T03:35:09.888 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_qfsck_on_mount": "true", 2023-10-20T03:35:09.888 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_retry_disk_reads": "3", 2023-10-20T03:35:09.889 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_rocksdb_cf": "true", 2023-10-20T03:35:09.889 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2023-10-20T03:35:09.889 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2023-10-20T03:35:09.889 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_rocksdb_options_annex": "", 2023-10-20T03:35:09.889 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_spdk_coremask": "0x1", 2023-10-20T03:35:09.890 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_spdk_io_sleep": "5", 2023-10-20T03:35:09.890 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_spdk_max_io_completion": "0", 2023-10-20T03:35:09.890 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_spdk_mem": "512", 2023-10-20T03:35:09.890 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_sync_submit_transaction": "false", 2023-10-20T03:35:09.890 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_bytes": "67108864", 2023-10-20T03:35:09.891 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_cost_per_io": "0", 2023-10-20T03:35:09.891 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2023-10-20T03:35:09.891 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2023-10-20T03:35:09.891 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2023-10-20T03:35:09.891 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_throttle_trace_rate": "0.000000", 2023-10-20T03:35:09.892 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_tracing": "false", 2023-10-20T03:35:09.892 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2023-10-20T03:35:09.892 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2023-10-20T03:35:09.892 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_volume_selection_reserved": "0", 2023-10-20T03:35:09.892 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2023-10-20T03:35:09.892 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2023-10-20T03:35:09.893 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_legacy_statfs": "true", 2023-10-20T03:35:09.893 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2023-10-20T03:35:09.893 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2023-10-20T03:35:09.893 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2023-10-20T03:35:09.893 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_zero_block_detection": "false", 2023-10-20T03:35:09.894 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm_path": "/usr/sbin/cephadm", 2023-10-20T03:35:09.894 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_action_update_interval": "2", 2023-10-20T03:35:09.894 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_directory_scan_interval": "10", 2023-10-20T03:35:09.894 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2023-10-20T03:35:09.894 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2023-10-20T03:35:09.895 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2023-10-20T03:35:09.895 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_mount_timeout": "10", 2023-10-20T03:35:09.895 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2023-10-20T03:35:09.895 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2023-10-20T03:35:09.895 INFO:teuthology.orchestra.run.smithi033.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2023-10-20T03:35:09.896 INFO:teuthology.orchestra.run.smithi033.stdout: "cephsqlite_blocklist_dead_locker": "true", 2023-10-20T03:35:09.896 INFO:teuthology.orchestra.run.smithi033.stdout: "cephsqlite_lock_renewal_interval": "2000", 2023-10-20T03:35:09.896 INFO:teuthology.orchestra.run.smithi033.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2023-10-20T03:35:09.896 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_cluster_require_signatures": "false", 2023-10-20T03:35:09.896 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_cluster_require_version": "2", 2023-10-20T03:35:09.897 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_require_signatures": "false", 2023-10-20T03:35:09.897 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_require_version": "2", 2023-10-20T03:35:09.897 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_service_require_signatures": "false", 2023-10-20T03:35:09.897 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_service_require_version": "2", 2023-10-20T03:35:09.897 INFO:teuthology.orchestra.run.smithi033.stdout: "cephx_sign_messages": "true", 2023-10-20T03:35:09.898 INFO:teuthology.orchestra.run.smithi033.stdout: "chdir": "", 2023-10-20T03:35:09.898 INFO:teuthology.orchestra.run.smithi033.stdout: "client_acl_type": "", 2023-10-20T03:35:09.898 INFO:teuthology.orchestra.run.smithi033.stdout: "client_asio_thread_count": "2", 2023-10-20T03:35:09.898 INFO:teuthology.orchestra.run.smithi033.stdout: "client_cache_mid": "0.750000", 2023-10-20T03:35:09.898 INFO:teuthology.orchestra.run.smithi033.stdout: "client_cache_size": "16384", 2023-10-20T03:35:09.899 INFO:teuthology.orchestra.run.smithi033.stdout: "client_caps_release_delay": "5", 2023-10-20T03:35:09.899 INFO:teuthology.orchestra.run.smithi033.stdout: "client_check_pool_perm": "true", 2023-10-20T03:35:09.899 INFO:teuthology.orchestra.run.smithi033.stdout: "client_collect_and_send_global_metrics": "false", 2023-10-20T03:35:09.899 INFO:teuthology.orchestra.run.smithi033.stdout: "client_debug_force_sync_read": "false", 2023-10-20T03:35:09.900 INFO:teuthology.orchestra.run.smithi033.stdout: "client_debug_getattr_caps": "false", 2023-10-20T03:35:09.900 INFO:teuthology.orchestra.run.smithi033.stdout: "client_debug_inject_tick_delay": "0", 2023-10-20T03:35:09.900 INFO:teuthology.orchestra.run.smithi033.stdout: "client_die_on_failed_dentry_invalidate": "true", 2023-10-20T03:35:09.900 INFO:teuthology.orchestra.run.smithi033.stdout: "client_die_on_failed_remount": "false", 2023-10-20T03:35:09.900 INFO:teuthology.orchestra.run.smithi033.stdout: "client_dirsize_rbytes": "true", 2023-10-20T03:35:09.900 INFO:teuthology.orchestra.run.smithi033.stdout: "client_force_lazyio": "false", 2023-10-20T03:35:09.901 INFO:teuthology.orchestra.run.smithi033.stdout: "client_fs": "", 2023-10-20T03:35:09.901 INFO:teuthology.orchestra.run.smithi033.stdout: "client_inject_fixed_oldest_tid": "false", 2023-10-20T03:35:09.901 INFO:teuthology.orchestra.run.smithi033.stdout: "client_inject_release_failure": "false", 2023-10-20T03:35:09.901 INFO:teuthology.orchestra.run.smithi033.stdout: "client_max_inline_size": "4096", 2023-10-20T03:35:09.901 INFO:teuthology.orchestra.run.smithi033.stdout: "client_max_retries_on_remount_failure": "5", 2023-10-20T03:35:09.902 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mds_namespace": "", 2023-10-20T03:35:09.902 INFO:teuthology.orchestra.run.smithi033.stdout: "client_metadata": "", 2023-10-20T03:35:09.902 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mount_gid": "-1", 2023-10-20T03:35:09.902 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mount_timeout": "300", 2023-10-20T03:35:09.902 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mount_uid": "-1", 2023-10-20T03:35:09.903 INFO:teuthology.orchestra.run.smithi033.stdout: "client_mountpoint": "/", 2023-10-20T03:35:09.903 INFO:teuthology.orchestra.run.smithi033.stdout: "client_notify_timeout": "10", 2023-10-20T03:35:09.903 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc": "true", 2023-10-20T03:35:09.903 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_max_dirty": "104857600", 2023-10-20T03:35:09.903 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_max_dirty_age": "5.000000", 2023-10-20T03:35:09.903 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_max_objects": "1000", 2023-10-20T03:35:09.904 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_size": "209715200", 2023-10-20T03:35:09.904 INFO:teuthology.orchestra.run.smithi033.stdout: "client_oc_target_dirty": "8388608", 2023-10-20T03:35:09.904 INFO:teuthology.orchestra.run.smithi033.stdout: "client_permissions": "true", 2023-10-20T03:35:09.904 INFO:teuthology.orchestra.run.smithi033.stdout: "client_quota": "true", 2023-10-20T03:35:09.904 INFO:teuthology.orchestra.run.smithi033.stdout: "client_quota_df": "true", 2023-10-20T03:35:09.904 INFO:teuthology.orchestra.run.smithi033.stdout: "client_readahead_max_bytes": "0", 2023-10-20T03:35:09.905 INFO:teuthology.orchestra.run.smithi033.stdout: "client_readahead_max_periods": "4", 2023-10-20T03:35:09.905 INFO:teuthology.orchestra.run.smithi033.stdout: "client_readahead_min": "131072", 2023-10-20T03:35:09.905 INFO:teuthology.orchestra.run.smithi033.stdout: "client_reconnect_stale": "false", 2023-10-20T03:35:09.905 INFO:teuthology.orchestra.run.smithi033.stdout: "client_shutdown_timeout": "30", 2023-10-20T03:35:09.905 INFO:teuthology.orchestra.run.smithi033.stdout: "client_snapdir": ".snap", 2023-10-20T03:35:09.905 INFO:teuthology.orchestra.run.smithi033.stdout: "client_tick_interval": "1", 2023-10-20T03:35:09.906 INFO:teuthology.orchestra.run.smithi033.stdout: "client_trace": "", 2023-10-20T03:35:09.906 INFO:teuthology.orchestra.run.smithi033.stdout: "client_try_dentry_invalidate": "false", 2023-10-20T03:35:09.906 INFO:teuthology.orchestra.run.smithi033.stdout: "client_use_faked_inos": "false", 2023-10-20T03:35:09.906 INFO:teuthology.orchestra.run.smithi033.stdout: "client_use_random_mds": "false", 2023-10-20T03:35:09.906 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_graylog": "false", 2023-10-20T03:35:09.907 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_graylog_host": "127.0.0.1", 2023-10-20T03:35:09.907 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_graylog_port": "12201", 2023-10-20T03:35:09.907 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_monitors": "default=true", 2023-10-20T03:35:09.907 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_syslog": "false", 2023-10-20T03:35:09.907 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2023-10-20T03:35:09.908 INFO:teuthology.orchestra.run.smithi033.stdout: "clog_to_syslog_level": "info", 2023-10-20T03:35:09.908 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "-", 2023-10-20T03:35:09.908 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_network": "", 2023-10-20T03:35:09.908 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_network_interface": "", 2023-10-20T03:35:09.908 INFO:teuthology.orchestra.run.smithi033.stdout: "compressor_zlib_isal": "false", 2023-10-20T03:35:09.908 INFO:teuthology.orchestra.run.smithi033.stdout: "compressor_zlib_level": "5", 2023-10-20T03:35:09.909 INFO:teuthology.orchestra.run.smithi033.stdout: "compressor_zlib_winsize": "-15", 2023-10-20T03:35:09.909 INFO:teuthology.orchestra.run.smithi033.stdout: "compressor_zstd_level": "1", 2023-10-20T03:35:09.909 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2023-10-20T03:35:09.909 INFO:teuthology.orchestra.run.smithi033.stdout: "crash_dir": "/var/lib/ceph/crash", 2023-10-20T03:35:09.909 INFO:teuthology.orchestra.run.smithi033.stdout: "crimson_alien_op_num_threads": "6", 2023-10-20T03:35:09.910 INFO:teuthology.orchestra.run.smithi033.stdout: "crimson_alien_thread_cpu_cores": "", 2023-10-20T03:35:09.910 INFO:teuthology.orchestra.run.smithi033.stdout: "crimson_osd_obc_lru_size": "10", 2023-10-20T03:35:09.910 INFO:teuthology.orchestra.run.smithi033.stdout: "crimson_osd_scheduler_concurrency": "0", 2023-10-20T03:35:09.910 INFO:teuthology.orchestra.run.smithi033.stdout: "crimson_seastar_smp": "1", 2023-10-20T03:35:09.911 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_location": "", 2023-10-20T03:35:09.911 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_location_hook": "", 2023-10-20T03:35:09.911 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_location_hook_timeout": "10", 2023-10-20T03:35:09.911 INFO:teuthology.orchestra.run.smithi033.stdout: "daemonize": "false", 2023-10-20T03:35:09.911 INFO:teuthology.orchestra.run.smithi033.stdout: "daos_pool": "tank", 2023-10-20T03:35:09.912 INFO:teuthology.orchestra.run.smithi033.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2023-10-20T03:35:09.912 INFO:teuthology.orchestra.run.smithi033.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2023-10-20T03:35:09.912 INFO:teuthology.orchestra.run.smithi033.stdout: "dbstore_db_name_prefix": "dbstore", 2023-10-20T03:35:09.912 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_alienstore": "0/5", 2023-10-20T03:35:09.912 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_allow_any_pool_priority": "false", 2023-10-20T03:35:09.912 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_asok": "1/5", 2023-10-20T03:35:09.913 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_asok_assert_abort": "false", 2023-10-20T03:35:09.913 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_asserts_on_shutdown": "true", 2023-10-20T03:35:09.913 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_auth": "20/20", 2023-10-20T03:35:09.913 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_bdev": "1/3", 2023-10-20T03:35:09.914 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_bluefs": "1/5", 2023-10-20T03:35:09.914 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_bluestore": "1/5", 2023-10-20T03:35:09.914 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_buffer": "0/1", 2023-10-20T03:35:09.914 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_ceph_exporter": "1/5", 2023-10-20T03:35:09.914 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_cephfs_mirror": "0/5", 2023-10-20T03:35:09.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_cephsqlite": "0/5", 2023-10-20T03:35:09.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_client": "0/5", 2023-10-20T03:35:09.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_compressor": "1/5", 2023-10-20T03:35:09.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_context": "0/1", 2023-10-20T03:35:09.915 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_crush": "1/1", 2023-10-20T03:35:09.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_crypto": "1/5", 2023-10-20T03:35:09.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_cyanstore": "0/5", 2023-10-20T03:35:09.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_deliberately_leak_memory": "false", 2023-10-20T03:35:09.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_disable_randomized_ping": "false", 2023-10-20T03:35:09.916 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_dpdk": "1/5", 2023-10-20T03:35:09.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_eventtrace": "1/5", 2023-10-20T03:35:09.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_filer": "0/1", 2023-10-20T03:35:09.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_filestore": "1/3", 2023-10-20T03:35:09.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_finisher": "1/1", 2023-10-20T03:35:09.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_fuse": "1/5", 2023-10-20T03:35:09.917 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_heartbeat_testing_span": "0", 2023-10-20T03:35:09.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_heartbeatmap": "1/5", 2023-10-20T03:35:09.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_immutable_obj_cache": "0/5", 2023-10-20T03:35:09.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_javaclient": "1/5", 2023-10-20T03:35:09.918 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_journal": "1/3", 2023-10-20T03:35:09.919 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_journaler": "0/5", 2023-10-20T03:35:09.919 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_kstore": "1/5", 2023-10-20T03:35:09.919 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_lockdep": "0/1", 2023-10-20T03:35:09.919 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mclock": "1/5", 2023-10-20T03:35:09.919 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds": "1/5", 2023-10-20T03:35:09.919 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_balancer": "1/5", 2023-10-20T03:35:09.920 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_locker": "1/5", 2023-10-20T03:35:09.920 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_log": "1/5", 2023-10-20T03:35:09.920 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_log_expire": "1/5", 2023-10-20T03:35:09.920 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mds_migrator": "1/5", 2023-10-20T03:35:09.921 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_memstore": "1/5", 2023-10-20T03:35:09.921 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mgr": "2/5", 2023-10-20T03:35:09.921 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mgrc": "1/5", 2023-10-20T03:35:09.921 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_mon": "20/20", 2023-10-20T03:35:09.921 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_monc": "0/10", 2023-10-20T03:35:09.922 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_ms": "1/1", 2023-10-20T03:35:09.922 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_none": "0/5", 2023-10-20T03:35:09.922 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_objclass": "0/5", 2023-10-20T03:35:09.922 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_objectcacher": "0/5", 2023-10-20T03:35:09.922 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_objecter": "0/1", 2023-10-20T03:35:09.923 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_optracker": "0/5", 2023-10-20T03:35:09.923 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_osd": "1/5", 2023-10-20T03:35:09.923 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_paxos": "20/20", 2023-10-20T03:35:09.923 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_perfcounter": "1/5", 2023-10-20T03:35:09.924 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_prioritycache": "1/5", 2023-10-20T03:35:09.924 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rados": "0/5", 2023-10-20T03:35:09.924 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rbd": "0/5", 2023-10-20T03:35:09.924 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rbd_mirror": "0/5", 2023-10-20T03:35:09.924 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rbd_pwl": "0/5", 2023-10-20T03:35:09.925 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rbd_replay": "0/5", 2023-10-20T03:35:09.925 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_refs": "0/0", 2023-10-20T03:35:09.925 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_reserver": "1/1", 2023-10-20T03:35:09.925 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw": "1/5", 2023-10-20T03:35:09.925 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw_access": "1/5", 2023-10-20T03:35:09.925 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw_datacache": "1/5", 2023-10-20T03:35:09.926 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw_dbstore": "1/5", 2023-10-20T03:35:09.926 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw_flight": "1/5", 2023-10-20T03:35:09.926 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rgw_sync": "1/5", 2023-10-20T03:35:09.926 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_rocksdb": "4/5", 2023-10-20T03:35:09.926 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore": "0/5", 2023-10-20T03:35:09.927 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_backref": "0/5", 2023-10-20T03:35:09.927 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_cache": "0/5", 2023-10-20T03:35:09.927 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_cleaner": "0/5", 2023-10-20T03:35:09.927 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_device": "0/5", 2023-10-20T03:35:09.927 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_epm": "0/5", 2023-10-20T03:35:09.928 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_fixedkv_tree": "0/5", 2023-10-20T03:35:09.928 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_journal": "0/5", 2023-10-20T03:35:09.928 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_lba": "0/5", 2023-10-20T03:35:09.928 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_odata": "0/5", 2023-10-20T03:35:09.928 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_omap": "0/5", 2023-10-20T03:35:09.928 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_onode": "0/5", 2023-10-20T03:35:09.929 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_t": "0/5", 2023-10-20T03:35:09.929 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_seastore_tm": "0/5", 2023-10-20T03:35:09.929 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_striper": "0/1", 2023-10-20T03:35:09.929 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_test": "0/5", 2023-10-20T03:35:09.929 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_throttle": "1/1", 2023-10-20T03:35:09.930 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_timer": "0/1", 2023-10-20T03:35:09.930 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_tp": "0/5", 2023-10-20T03:35:09.930 INFO:teuthology.orchestra.run.smithi033.stdout: "debug_trace": "1/5", 2023-10-20T03:35:09.930 INFO:teuthology.orchestra.run.smithi033.stdout: "defer_client_eviction_on_laggy_osds": "true", 2023-10-20T03:35:09.930 INFO:teuthology.orchestra.run.smithi033.stdout: "device_failure_prediction_mode": "none", 2023-10-20T03:35:09.931 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2023-10-20T03:35:09.931 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2023-10-20T03:35:09.931 INFO:teuthology.orchestra.run.smithi033.stdout: "err_to_graylog": "false", 2023-10-20T03:35:09.931 INFO:teuthology.orchestra.run.smithi033.stdout: "err_to_journald": "false", 2023-10-20T03:35:09.931 INFO:teuthology.orchestra.run.smithi033.stdout: "err_to_stderr": "true", 2023-10-20T03:35:09.932 INFO:teuthology.orchestra.run.smithi033.stdout: "err_to_syslog": "false", 2023-10-20T03:35:09.932 INFO:teuthology.orchestra.run.smithi033.stdout: "event_tracing": "false", 2023-10-20T03:35:09.932 INFO:teuthology.orchestra.run.smithi033.stdout: "exporter_addr": "0.0.0.0", 2023-10-20T03:35:09.932 INFO:teuthology.orchestra.run.smithi033.stdout: "exporter_http_port": "9926", 2023-10-20T03:35:09.932 INFO:teuthology.orchestra.run.smithi033.stdout: "exporter_prio_limit": "5", 2023-10-20T03:35:09.933 INFO:teuthology.orchestra.run.smithi033.stdout: "exporter_sock_dir": "/var/run/ceph/", 2023-10-20T03:35:09.933 INFO:teuthology.orchestra.run.smithi033.stdout: "exporter_sort_metrics": "true", 2023-10-20T03:35:09.933 INFO:teuthology.orchestra.run.smithi033.stdout: "exporter_stats_period": "5", 2023-10-20T03:35:09.933 INFO:teuthology.orchestra.run.smithi033.stdout: "fake_statfs_for_testing": "0", 2023-10-20T03:35:09.933 INFO:teuthology.orchestra.run.smithi033.stdout: "fatal_signal_handlers": "true", 2023-10-20T03:35:09.934 INFO:teuthology.orchestra.run.smithi033.stdout: "filer_max_purge_ops": "10", 2023-10-20T03:35:09.934 INFO:teuthology.orchestra.run.smithi033.stdout: "filer_max_truncate_ops": "128", 2023-10-20T03:35:09.934 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_apply_finisher_threads": "1", 2023-10-20T03:35:09.934 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_blackhole": "false", 2023-10-20T03:35:09.934 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_btrfs_clone_range": "true", 2023-10-20T03:35:09.935 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_btrfs_snap": "true", 2023-10-20T03:35:09.935 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_caller_concurrency": "10", 2023-10-20T03:35:09.935 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_collect_device_partition_information": "true", 2023-10-20T03:35:09.935 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_commit_timeout": "600.000000", 2023-10-20T03:35:09.935 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_debug_inject_read_err": "false", 2023-10-20T03:35:09.935 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_debug_omap_check": "false", 2023-10-20T03:35:09.936 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_debug_random_read_err": "0.000000", 2023-10-20T03:35:09.936 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_debug_verify_split": "false", 2023-10-20T03:35:09.936 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_dump_file": "", 2023-10-20T03:35:09.936 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2023-10-20T03:35:09.937 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_expected_throughput_ops": "200.000000", 2023-10-20T03:35:09.937 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fadvise": "true", 2023-10-20T03:35:09.937 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fail_eio": "true", 2023-10-20T03:35:09.937 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fd_cache_shards": "16", 2023-10-20T03:35:09.937 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fd_cache_size": "128", 2023-10-20T03:35:09.937 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fiemap": "false", 2023-10-20T03:35:09.938 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fiemap_threshold": "4096", 2023-10-20T03:35:09.938 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_fsync_flushes_journal_data": "false", 2023-10-20T03:35:09.938 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_index_retry_probability": "0.000000", 2023-10-20T03:35:09.938 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_inject_stall": "0", 2023-10-20T03:35:09.939 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_journal_parallel": "false", 2023-10-20T03:35:09.939 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_journal_trailing": "false", 2023-10-20T03:35:09.939 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_journal_writeahead": "false", 2023-10-20T03:35:09.939 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_kill_at": "0", 2023-10-20T03:35:09.939 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_alloc_hint_size": "1048576", 2023-10-20T03:35:09.939 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattr_size": "0", 2023-10-20T03:35:09.940 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2023-10-20T03:35:09.940 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattr_size_other": "512", 2023-10-20T03:35:09.940 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2023-10-20T03:35:09.940 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattrs": "0", 2023-10-20T03:35:09.940 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2023-10-20T03:35:09.941 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattrs_other": "2", 2023-10-20T03:35:09.941 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_inline_xattrs_xfs": "10", 2023-10-20T03:35:09.941 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_sync_interval": "5.000000", 2023-10-20T03:35:09.941 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_xattr_value_size": "0", 2023-10-20T03:35:09.941 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2023-10-20T03:35:09.942 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_xattr_value_size_other": "1024", 2023-10-20T03:35:09.942 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2023-10-20T03:35:09.942 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_merge_threshold": "-10", 2023-10-20T03:35:09.942 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_min_sync_interval": "0.010000", 2023-10-20T03:35:09.942 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_odsync_write": "false", 2023-10-20T03:35:09.943 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_omap_backend": "rocksdb", 2023-10-20T03:35:09.943 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_omap_backend_path": "", 2023-10-20T03:35:09.943 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_omap_header_cache_size": "1024", 2023-10-20T03:35:09.943 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_ondisk_finisher_threads": "1", 2023-10-20T03:35:09.943 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_op_thread_suicide_timeout": "180", 2023-10-20T03:35:09.943 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_op_thread_timeout": "60", 2023-10-20T03:35:09.944 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_op_threads": "2", 2023-10-20T03:35:09.944 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_punch_hole": "false", 2023-10-20T03:35:09.944 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2023-10-20T03:35:09.944 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2023-10-20T03:35:09.945 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2023-10-20T03:35:09.945 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_high_threshhold": "0.900000", 2023-10-20T03:35:09.945 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_low_threshhold": "0.300000", 2023-10-20T03:35:09.945 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_bytes": "104857600", 2023-10-20T03:35:09.945 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2023-10-20T03:35:09.945 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2023-10-20T03:35:09.946 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2023-10-20T03:35:09.946 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_queue_max_ops": "50", 2023-10-20T03:35:09.946 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2023-10-20T03:35:09.946 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_seek_data_hole": "false", 2023-10-20T03:35:09.946 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_sloppy_crc": "false", 2023-10-20T03:35:09.947 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_sloppy_crc_block_size": "65536", 2023-10-20T03:35:09.947 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_splice": "false", 2023-10-20T03:35:09.947 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_split_multiple": "2", 2023-10-20T03:35:09.947 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_split_rand_factor": "20", 2023-10-20T03:35:09.947 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_update_to": "1000", 2023-10-20T03:35:09.948 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2023-10-20T03:35:09.948 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2023-10-20T03:35:09.948 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2023-10-20T03:35:09.948 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2023-10-20T03:35:09.948 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2023-10-20T03:35:09.949 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2023-10-20T03:35:09.949 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_enable": "true", 2023-10-20T03:35:09.949 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2023-10-20T03:35:09.949 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2023-10-20T03:35:09.949 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2023-10-20T03:35:09.949 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2023-10-20T03:35:09.950 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2023-10-20T03:35:09.950 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2023-10-20T03:35:09.950 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_xfs_extsize": "false", 2023-10-20T03:35:09.950 INFO:teuthology.orchestra.run.smithi033.stdout: "filestore_zfs_snap": "false", 2023-10-20T03:35:09.950 INFO:teuthology.orchestra.run.smithi033.stdout: "fio_dir": "/tmp/fio", 2023-10-20T03:35:09.951 INFO:teuthology.orchestra.run.smithi033.stdout: "fsid": "b920f165-e9e7-4c3e-8d7b-cfdc002ae589", 2023-10-20T03:35:09.951 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_allow_other": "true", 2023-10-20T03:35:09.951 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_atomic_o_trunc": "true", 2023-10-20T03:35:09.951 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_big_writes": "true", 2023-10-20T03:35:09.953 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_debug": "false", 2023-10-20T03:35:09.954 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_default_permissions": "false", 2023-10-20T03:35:09.954 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_disable_pagecache": "false", 2023-10-20T03:35:09.954 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_max_write": "0", 2023-10-20T03:35:09.954 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_multithreaded": "true", 2023-10-20T03:35:09.954 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_require_active_mds": "true", 2023-10-20T03:35:09.955 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_set_user_groups": "true", 2023-10-20T03:35:09.955 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_splice_move": "true", 2023-10-20T03:35:09.955 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_splice_read": "true", 2023-10-20T03:35:09.955 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_splice_write": "true", 2023-10-20T03:35:09.955 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_syncfs_on_mksnap": "true", 2023-10-20T03:35:09.956 INFO:teuthology.orchestra.run.smithi033.stdout: "fuse_use_invalidate_cb": "true", 2023-10-20T03:35:09.956 INFO:teuthology.orchestra.run.smithi033.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2023-10-20T03:35:09.956 INFO:teuthology.orchestra.run.smithi033.stdout: "gss_target_name": "ceph", 2023-10-20T03:35:09.956 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_file": "", 2023-10-20T03:35:09.956 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_inject_failure": "0", 2023-10-20T03:35:09.957 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_interval": "5", 2023-10-20T03:35:09.957 INFO:teuthology.orchestra.run.smithi033.stdout: "host": "smithi033", 2023-10-20T03:35:09.957 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2023-10-20T03:35:09.957 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_max_inflight_ops": "128", 2023-10-20T03:35:09.957 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_max_size": "1073741824", 2023-10-20T03:35:09.958 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2023-10-20T03:35:09.958 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_bps_burst": "0", 2023-10-20T03:35:09.958 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2023-10-20T03:35:09.958 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_bps_limit": "0", 2023-10-20T03:35:09.958 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_iops_burst": "0", 2023-10-20T03:35:09.959 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2023-10-20T03:35:09.959 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_iops_limit": "0", 2023-10-20T03:35:09.959 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2023-10-20T03:35:09.959 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2023-10-20T03:35:09.959 INFO:teuthology.orchestra.run.smithi033.stdout: "immutable_object_cache_watermark": "0.900000", 2023-10-20T03:35:09.959 INFO:teuthology.orchestra.run.smithi033.stdout: "inject_early_sigterm": "false", 2023-10-20T03:35:09.960 INFO:teuthology.orchestra.run.smithi033.stdout: "jaeger_agent_port": "6799", 2023-10-20T03:35:09.960 INFO:teuthology.orchestra.run.smithi033.stdout: "jaeger_tracing_enable": "false", 2023-10-20T03:35:09.960 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_aio": "true", 2023-10-20T03:35:09.960 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_align_min_size": "65536", 2023-10-20T03:35:09.960 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_block_align": "true", 2023-10-20T03:35:09.961 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_block_size": "4096", 2023-10-20T03:35:09.961 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_dio": "true", 2023-10-20T03:35:09.961 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_discard": "false", 2023-10-20T03:35:09.961 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_force_aio": "false", 2023-10-20T03:35:09.961 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_ignore_corruption": "false", 2023-10-20T03:35:09.962 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_max_write_bytes": "10485760", 2023-10-20T03:35:09.962 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_max_write_entries": "100", 2023-10-20T03:35:09.962 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_replay_from": "0", 2023-10-20T03:35:09.962 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_throttle_high_multiple": "0.000000", 2023-10-20T03:35:09.962 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_throttle_high_threshhold": "0.900000", 2023-10-20T03:35:09.962 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_throttle_low_threshhold": "0.600000", 2023-10-20T03:35:09.963 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_throttle_max_multiple": "0.000000", 2023-10-20T03:35:09.963 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_write_header_frequency": "0", 2023-10-20T03:35:09.963 INFO:teuthology.orchestra.run.smithi033.stdout: "journal_zero_on_create": "false", 2023-10-20T03:35:09.963 INFO:teuthology.orchestra.run.smithi033.stdout: "journaler_prefetch_periods": "10", 2023-10-20T03:35:09.964 INFO:teuthology.orchestra.run.smithi033.stdout: "journaler_prezero_periods": "5", 2023-10-20T03:35:09.964 INFO:teuthology.orchestra.run.smithi033.stdout: "journaler_write_head_interval": "15", 2023-10-20T03:35:09.964 INFO:teuthology.orchestra.run.smithi033.stdout: "key": "", 2023-10-20T03:35:09.964 INFO:teuthology.orchestra.run.smithi033.stdout: "keyfile": "", 2023-10-20T03:35:09.964 INFO:teuthology.orchestra.run.smithi033.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2023-10-20T03:35:09.964 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_backend": "rocksdb", 2023-10-20T03:35:09.965 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_default_stripe_size": "65536", 2023-10-20T03:35:09.965 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_fsck_on_mount": "false", 2023-10-20T03:35:09.965 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_fsck_on_mount_deep": "true", 2023-10-20T03:35:09.965 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_max_bytes": "67108864", 2023-10-20T03:35:09.965 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_max_ops": "512", 2023-10-20T03:35:09.966 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_nid_prealloc": "1024", 2023-10-20T03:35:09.966 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_onode_map_size": "1024", 2023-10-20T03:35:09.966 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2023-10-20T03:35:09.966 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_sync_submit_transaction": "false", 2023-10-20T03:35:09.966 INFO:teuthology.orchestra.run.smithi033.stdout: "kstore_sync_transaction": "false", 2023-10-20T03:35:09.967 INFO:teuthology.orchestra.run.smithi033.stdout: "librados_thread_count": "2", 2023-10-20T03:35:09.967 INFO:teuthology.orchestra.run.smithi033.stdout: "lockdep": "false", 2023-10-20T03:35:09.967 INFO:teuthology.orchestra.run.smithi033.stdout: "lockdep_force_backtrace": "false", 2023-10-20T03:35:09.967 INFO:teuthology.orchestra.run.smithi033.stdout: "log_coarse_timestamps": "true", 2023-10-20T03:35:09.967 INFO:teuthology.orchestra.run.smithi033.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2023-10-20T03:35:09.967 INFO:teuthology.orchestra.run.smithi033.stdout: "log_flush_on_exit": "false", 2023-10-20T03:35:09.968 INFO:teuthology.orchestra.run.smithi033.stdout: "log_graylog_host": "127.0.0.1", 2023-10-20T03:35:09.968 INFO:teuthology.orchestra.run.smithi033.stdout: "log_graylog_port": "12201", 2023-10-20T03:35:09.968 INFO:teuthology.orchestra.run.smithi033.stdout: "log_max_new": "1000", 2023-10-20T03:35:09.968 INFO:teuthology.orchestra.run.smithi033.stdout: "log_max_recent": "10000", 2023-10-20T03:35:09.968 INFO:teuthology.orchestra.run.smithi033.stdout: "log_stderr_prefix": "", 2023-10-20T03:35:09.969 INFO:teuthology.orchestra.run.smithi033.stdout: "log_stop_at_utilization": "0.970000", 2023-10-20T03:35:09.969 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": "true", 2023-10-20T03:35:09.969 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_graylog": "false", 2023-10-20T03:35:09.969 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_journald": "false", 2023-10-20T03:35:09.969 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_stderr": "false", 2023-10-20T03:35:09.969 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_syslog": "false", 2023-10-20T03:35:09.970 INFO:teuthology.orchestra.run.smithi033.stdout: "max_rotating_auth_attempts": "10", 2023-10-20T03:35:09.970 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2023-10-20T03:35:09.970 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_action_on_write_error": "1", 2023-10-20T03:35:09.970 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_alternate_name_max": "8192", 2023-10-20T03:35:09.970 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_asio_thread_count": "2", 2023-10-20T03:35:09.971 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_export_pin": "true", 2023-10-20T03:35:09.971 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_fragment_dirs": "true", 2023-10-20T03:35:09.971 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2023-10-20T03:35:09.971 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_fragment_interval": "5", 2023-10-20T03:35:09.971 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_fragment_size_max": "100000", 2023-10-20T03:35:09.971 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_idle_threshold": "0.000000", 2023-10-20T03:35:09.972 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_interval": "10", 2023-10-20T03:35:09.972 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_max": "-1", 2023-10-20T03:35:09.972 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_max_until": "-1", 2023-10-20T03:35:09.972 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_merge_size": "50", 2023-10-20T03:35:09.972 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_midchunk": "0.300000", 2023-10-20T03:35:09.972 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_min_rebalance": "0.100000", 2023-10-20T03:35:09.973 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_min_start": "0.200000", 2023-10-20T03:35:09.973 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_minchunk": "0.001000", 2023-10-20T03:35:09.973 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_mode": "0", 2023-10-20T03:35:09.973 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_need_max": "1.200000", 2023-10-20T03:35:09.973 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_need_min": "0.800000", 2023-10-20T03:35:09.974 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_overload_epochs": "2", 2023-10-20T03:35:09.974 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_replicate_threshold": "8000.000000", 2023-10-20T03:35:09.974 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_sample_interval": "3.000000", 2023-10-20T03:35:09.974 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_split_bits": "3", 2023-10-20T03:35:09.974 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_split_rd": "25000.000000", 2023-10-20T03:35:09.975 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_split_size": "10000", 2023-10-20T03:35:09.975 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_split_wr": "10000.000000", 2023-10-20T03:35:09.975 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_target_decay": "10.000000", 2023-10-20T03:35:09.975 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2023-10-20T03:35:09.975 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_beacon_grace": "15.000000", 2023-10-20T03:35:09.976 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_beacon_interval": "4.000000", 2023-10-20T03:35:09.976 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_beacon_mon_down_grace": "60", 2023-10-20T03:35:09.976 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_memory_limit": "4294967296", 2023-10-20T03:35:09.976 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_mid": "0.700000", 2023-10-20T03:35:09.977 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_release_free_interval": "10", 2023-10-20T03:35:09.977 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_reservation": "0.050000", 2023-10-20T03:35:09.977 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_trim_decay_rate": "1.000000", 2023-10-20T03:35:09.977 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_trim_interval": "1", 2023-10-20T03:35:09.977 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cache_trim_threshold": "262144", 2023-10-20T03:35:09.977 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2023-10-20T03:35:09.978 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2023-10-20T03:35:09.978 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_client_delegate_inos_pct": "50", 2023-10-20T03:35:09.978 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_client_prealloc_inos": "1000", 2023-10-20T03:35:09.978 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2023-10-20T03:35:09.979 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_connect_bootstrapping": "false", 2023-10-20T03:35:09.979 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_damage_table_max_entries": "10000", 2023-10-20T03:35:09.979 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2023-10-20T03:35:09.979 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_debug_auth_pins": "false", 2023-10-20T03:35:09.979 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_debug_frag": "false", 2023-10-20T03:35:09.980 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_debug_scatterstat": "false", 2023-10-20T03:35:09.980 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_debug_subtrees": "false", 2023-10-20T03:35:09.980 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_decay_halflife": "5.000000", 2023-10-20T03:35:09.980 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_default_dir_hash": "2", 2023-10-20T03:35:09.980 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_defer_session_stale": "true", 2023-10-20T03:35:09.981 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_deny_all_reconnect": "false", 2023-10-20T03:35:09.981 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dir_keys_per_op": "16384", 2023-10-20T03:35:09.981 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dir_max_commit_size": "10", 2023-10-20T03:35:09.981 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dir_max_entries": "0", 2023-10-20T03:35:09.982 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dir_prefetch": "true", 2023-10-20T03:35:09.982 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dirstat_min_interval": "1.000000", 2023-10-20T03:35:09.982 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dump_cache_after_rejoin": "false", 2023-10-20T03:35:09.982 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dump_cache_on_map": "false", 2023-10-20T03:35:09.983 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dump_cache_threshold_file": "0", 2023-10-20T03:35:09.983 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2023-10-20T03:35:09.983 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_early_reply": "true", 2023-10-20T03:35:09.984 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_enable_op_tracker": "true", 2023-10-20T03:35:09.984 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_enforce_unique_name": "true", 2023-10-20T03:35:09.984 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_export_ephemeral_distributed": "true", 2023-10-20T03:35:09.984 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2023-10-20T03:35:09.985 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_export_ephemeral_random": "true", 2023-10-20T03:35:09.985 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_export_ephemeral_random_max": "0.010000", 2023-10-20T03:35:09.985 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_extraordinary_events_dump_interval": "0", 2023-10-20T03:35:09.985 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_forward_all_requests_to_auth": "false", 2023-10-20T03:35:09.985 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_freeze_tree_timeout": "30.000000", 2023-10-20T03:35:09.986 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_fscrypt_last_block_max_size": "4096", 2023-10-20T03:35:09.986 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_go_bad_corrupt_dentry": "true", 2023-10-20T03:35:09.986 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2023-10-20T03:35:09.986 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_health_cache_threshold": "1.500000", 2023-10-20T03:35:09.986 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_health_summarize_threshold": "10", 2023-10-20T03:35:09.987 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_heartbeat_grace": "15.000000", 2023-10-20T03:35:09.987 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_heartbeat_reset_grace": "1000", 2023-10-20T03:35:09.987 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_health_dummy": "false", 2023-10-20T03:35:09.987 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2023-10-20T03:35:09.988 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_migrator_session_race": "false", 2023-10-20T03:35:09.988 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2023-10-20T03:35:09.988 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_skip_replaying_inotable": "false", 2023-10-20T03:35:09.988 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2023-10-20T03:35:09.988 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_join_fs": "", 2023-10-20T03:35:09.989 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_journal_format": "1", 2023-10-20T03:35:09.989 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_create_at": "0", 2023-10-20T03:35:09.989 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_export_at": "0", 2023-10-20T03:35:09.989 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_import_at": "0", 2023-10-20T03:35:09.989 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_journal_at": "0", 2023-10-20T03:35:09.990 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_journal_expire_at": "0", 2023-10-20T03:35:09.990 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_journal_replay_at": "0", 2023-10-20T03:35:09.990 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_link_at": "0", 2023-10-20T03:35:09.991 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_mdstable_at": "0", 2023-10-20T03:35:09.991 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_openc_at": "0", 2023-10-20T03:35:09.991 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_rename_at": "0", 2023-10-20T03:35:09.992 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_shutdown_at": "0", 2023-10-20T03:35:09.992 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_kill_skip_replaying_inotable": "false", 2023-10-20T03:35:09.992 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_event_large_threshold": "524288", 2023-10-20T03:35:09.992 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_events_per_segment": "1024", 2023-10-20T03:35:09.992 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_major_segment_event_ratio": "12", 2023-10-20T03:35:09.993 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_max_events": "-1", 2023-10-20T03:35:09.993 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_max_segments": "128", 2023-10-20T03:35:09.993 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_pause": "false", 2023-10-20T03:35:09.993 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_segment_size": "0", 2023-10-20T03:35:09.994 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_skip_corrupt_events": "false", 2023-10-20T03:35:09.994 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_skip_unbounded_events": "false", 2023-10-20T03:35:09.994 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_log_warn_factor": "2.000000", 2023-10-20T03:35:09.994 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_caps_per_client": "1048576", 2023-10-20T03:35:09.994 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_completed_flushes": "100000", 2023-10-20T03:35:09.995 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_completed_requests": "100000", 2023-10-20T03:35:09.995 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_export_size": "20971520", 2023-10-20T03:35:09.995 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_file_recover": "32", 2023-10-20T03:35:09.995 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_purge_files": "64", 2023-10-20T03:35:09.996 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_purge_ops": "8192", 2023-10-20T03:35:09.996 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2023-10-20T03:35:09.996 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_scrub_ops_in_progress": "5", 2023-10-20T03:35:09.996 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_max_snaps_per_dir": "100", 2023-10-20T03:35:09.996 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_metrics_update_interval": "2", 2023-10-20T03:35:09.997 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_min_caps_per_client": "100", 2023-10-20T03:35:09.997 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_min_caps_working_set": "10000", 2023-10-20T03:35:09.997 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_mon_shutdown_timeout": "5.000000", 2023-10-20T03:35:09.997 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_numa_node": "-1", 2023-10-20T03:35:09.997 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_oft_prefetch_dirfrags": "false", 2023-10-20T03:35:09.998 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_complaint_time": "30.000000", 2023-10-20T03:35:09.998 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_history_duration": "600", 2023-10-20T03:35:09.998 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_history_size": "20", 2023-10-20T03:35:09.998 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_history_slow_op_size": "20", 2023-10-20T03:35:09.998 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_history_slow_op_threshold": "10", 2023-10-20T03:35:09.999 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_op_log_threshold": "5", 2023-10-20T03:35:09.999 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_ping_grace": "15", 2023-10-20T03:35:09.999 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_ping_interval": "5", 2023-10-20T03:35:09.999 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2023-10-20T03:35:10.000 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_global_max_decay_threshold": "131072", 2023-10-20T03:35:10.000 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_max_caps": "30000", 2023-10-20T03:35:10.000 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_max_decay_rate": "1.500000", 2023-10-20T03:35:10.000 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_max_decay_threshold": "131072", 2023-10-20T03:35:10.000 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_warning_decay_rate": "60.000000", 2023-10-20T03:35:10.001 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_recall_warning_threshold": "262144", 2023-10-20T03:35:10.001 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_reconnect_timeout": "45.000000", 2023-10-20T03:35:10.001 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_replay_interval": "1.000000", 2023-10-20T03:35:10.001 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_replay_unsafe_with_closed_session": "false", 2023-10-20T03:35:10.001 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_request_load_average_decay_rate": "60.000000", 2023-10-20T03:35:10.002 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_root_ino_gid": "0", 2023-10-20T03:35:10.002 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_root_ino_uid": "0", 2023-10-20T03:35:10.002 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_scatter_nudge_interval": "5.000000", 2023-10-20T03:35:10.002 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_blocklist_on_evict": "true", 2023-10-20T03:35:10.003 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_blocklist_on_timeout": "true", 2023-10-20T03:35:10.003 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2023-10-20T03:35:10.003 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_cache_liveness_magnitude": "10", 2023-10-20T03:35:10.003 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2023-10-20T03:35:10.003 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_cap_acquisition_throttle": "100000", 2023-10-20T03:35:10.004 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2023-10-20T03:35:10.004 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_session_metadata_threshold": "16777216", 2023-10-20T03:35:10.004 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sessionmap_keys_per_op": "1024", 2023-10-20T03:35:10.004 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_shutdown_check": "0", 2023-10-20T03:35:10.004 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_skip_ino": "0", 2023-10-20T03:35:10.005 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_sleep_rank_change": "0.000000", 2023-10-20T03:35:10.005 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_snap_max_uid": "4294967294", 2023-10-20T03:35:10.005 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_snap_min_uid": "0", 2023-10-20T03:35:10.005 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_snap_rstat": "false", 2023-10-20T03:35:10.006 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_standby_replay_damaged": "false", 2023-10-20T03:35:10.006 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_symlink_recovery": "true", 2023-10-20T03:35:10.006 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_task_status_update_interval": "2.000000", 2023-10-20T03:35:10.006 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_thrash_exports": "0", 2023-10-20T03:35:10.006 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_thrash_fragments": "0", 2023-10-20T03:35:10.007 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_tick_interval": "5.000000", 2023-10-20T03:35:10.007 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_valgrind_exit": "false", 2023-10-20T03:35:10.007 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_verify_backtrace": "1", 2023-10-20T03:35:10.007 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_verify_scatter": "false", 2023-10-20T03:35:10.008 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_wipe_ino_prealloc": "false", 2023-10-20T03:35:10.008 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_wipe_sessions": "false", 2023-10-20T03:35:10.008 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool_debug": "false", 2023-10-20T03:35:10.009 INFO:teuthology.orchestra.run.smithi033.stdout: "memstore_debug_omit_block_device_write": "false", 2023-10-20T03:35:10.009 INFO:teuthology.orchestra.run.smithi033.stdout: "memstore_device_bytes": "1073741824", 2023-10-20T03:35:10.009 INFO:teuthology.orchestra.run.smithi033.stdout: "memstore_page_set": "false", 2023-10-20T03:35:10.009 INFO:teuthology.orchestra.run.smithi033.stdout: "memstore_page_size": "65536", 2023-10-20T03:35:10.009 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_client_bytes": "134217728", 2023-10-20T03:35:10.010 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_client_messages": "512", 2023-10-20T03:35:10.010 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2023-10-20T03:35:10.011 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_connect_retry_interval": "1.000000", 2023-10-20T03:35:10.011 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2023-10-20T03:35:10.012 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2023-10-20T03:35:10.012 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_disabled_modules": "", 2023-10-20T03:35:10.012 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_initial_modules": "restful iostat nfs", 2023-10-20T03:35:10.013 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_max_pg_num_change": "128", 2023-10-20T03:35:10.013 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_mds_bytes": "134217728", 2023-10-20T03:35:10.013 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_mds_messages": "128", 2023-10-20T03:35:10.014 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2023-10-20T03:35:10.014 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_mon_bytes": "134217728", 2023-10-20T03:35:10.014 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_mon_messages": "128", 2023-10-20T03:35:10.014 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_osd_bytes": "536870912", 2023-10-20T03:35:10.015 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_osd_messages": "8192", 2023-10-20T03:35:10.015 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_pool": "true", 2023-10-20T03:35:10.016 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_service_beacon_grace": "60.000000", 2023-10-20T03:35:10.016 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_standby_modules": "true", 2023-10-20T03:35:10.016 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_stats_period": "5", 2023-10-20T03:35:10.016 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_stats_threshold": "5", 2023-10-20T03:35:10.016 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_tick_period": "2", 2023-10-20T03:35:10.017 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_ttl_cache_expire_seconds": "0", 2023-10-20T03:35:10.017 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_accept_timeout_factor": "2.000000", 2023-10-20T03:35:10.017 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_allow_pool_delete": "true", 2023-10-20T03:35:10.017 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_allow_pool_size_one": "true", 2023-10-20T03:35:10.018 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_auth_validate_all_caps": "true", 2023-10-20T03:35:10.018 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2023-10-20T03:35:10.018 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2023-10-20T03:35:10.018 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_bytes": "104857600", 2023-10-20T03:35:10.019 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_directed_command_retry": "5", 2023-10-20T03:35:10.019 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_interval": "3.000000", 2023-10-20T03:35:10.019 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2023-10-20T03:35:10.019 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2023-10-20T03:35:10.020 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2023-10-20T03:35:10.020 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_hunt_parallel": "3", 2023-10-20T03:35:10.020 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_log_interval": "1.000000", 2023-10-20T03:35:10.020 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_max_log_entries_per_message": "1000", 2023-10-20T03:35:10.021 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_ping_interval": "10.000000", 2023-10-20T03:35:10.021 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_ping_timeout": "30.000000", 2023-10-20T03:35:10.021 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_client_target_rank": "-1", 2023-10-20T03:35:10.021 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_clock_drift_allowed": "1.000000", 2023-10-20T03:35:10.021 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2023-10-20T03:35:10.022 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2023-10-20T03:35:10.022 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_file_level": "debug", 2023-10-20T03:35:10.022 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_file": "true", 2023-10-20T03:35:10.022 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_graylog": "false", 2023-10-20T03:35:10.022 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2023-10-20T03:35:10.023 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_graylog_port": "12201", 2023-10-20T03:35:10.023 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_journald": "false", 2023-10-20T03:35:10.023 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_stderr": "false", 2023-10-20T03:35:10.023 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_syslog": "default=false", 2023-10-20T03:35:10.023 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2023-10-20T03:35:10.024 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cluster_log_to_syslog_level": "info", 2023-10-20T03:35:10.024 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_compact_on_bootstrap": "false", 2023-10-20T03:35:10.024 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_compact_on_start": "false", 2023-10-20T03:35:10.024 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_compact_on_trim": "true", 2023-10-20T03:35:10.024 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_con_tracker_persist_interval": "10", 2023-10-20T03:35:10.025 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_con_tracker_score_halflife": "43200", 2023-10-20T03:35:10.025 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_config_key_max_entry_size": "65536", 2023-10-20T03:35:10.025 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_cpu_threads": "4", 2023-10-20T03:35:10.025 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_crush_min_required_version": "hammer", 2023-10-20T03:35:10.025 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_daemon_bytes": "419430400", 2023-10-20T03:35:10.026 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2023-10-20T03:35:10.026 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_data_avail_crit": "5", 2023-10-20T03:35:10.026 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_data_avail_warn": "5", 2023-10-20T03:35:10.026 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_data_size_warn": "16106127360", 2023-10-20T03:35:10.027 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_block_osdmap_trim": "false", 2023-10-20T03:35:10.027 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_deprecated_as_obsolete": "false", 2023-10-20T03:35:10.027 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_dump_json": "false", 2023-10-20T03:35:10.027 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2023-10-20T03:35:10.028 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_dump_transactions": "false", 2023-10-20T03:35:10.028 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_extra_checks": "false", 2023-10-20T03:35:10.028 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_no_initial_persistent_features": "false", 2023-10-20T03:35:10.028 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2023-10-20T03:35:10.028 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_no_require_quincy": "false", 2023-10-20T03:35:10.029 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_no_require_reef": "false", 2023-10-20T03:35:10.029 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2023-10-20T03:35:10.029 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_delta_reset_interval": "10.000000", 2023-10-20T03:35:10.029 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_dns_srv_name": "ceph-mon", 2023-10-20T03:35:10.030 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_down_mkfs_grace": "120", 2023-10-20T03:35:10.030 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_election_default_strategy": "1", 2023-10-20T03:35:10.030 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_election_timeout": "5.000000", 2023-10-20T03:35:10.030 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2023-10-20T03:35:10.030 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_elector_ping_divisor": "2", 2023-10-20T03:35:10.031 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_elector_ping_timeout": "2.000000", 2023-10-20T03:35:10.031 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_enable_op_tracker": "true", 2023-10-20T03:35:10.031 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_fake_pool_delete": "false", 2023-10-20T03:35:10.031 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_force_quorum_join": "false", 2023-10-20T03:35:10.031 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_globalid_prealloc": "10000", 2023-10-20T03:35:10.032 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_detail_to_clog": "false", 2023-10-20T03:35:10.032 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_log_update_period": "5", 2023-10-20T03:35:10.032 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_max_detail": "50", 2023-10-20T03:35:10.032 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_to_clog": "true", 2023-10-20T03:35:10.032 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_to_clog_interval": "600", 2023-10-20T03:35:10.033 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2023-10-20T03:35:10.033 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_host": "172.21.15.33,172.21.15.40,172.21.15.64", 2023-10-20T03:35:10.033 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_host_override": "", 2023-10-20T03:35:10.033 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_initial_members": "", 2023-10-20T03:35:10.033 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2023-10-20T03:35:10.034 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2023-10-20T03:35:10.034 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_inject_transaction_delay_max": "10.000000", 2023-10-20T03:35:10.034 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2023-10-20T03:35:10.034 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_keyvaluedb": "rocksdb", 2023-10-20T03:35:10.035 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_lease": "5.000000", 2023-10-20T03:35:10.035 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2023-10-20T03:35:10.035 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_lease_renew_interval_factor": "0.600000", 2023-10-20T03:35:10.035 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_log_full_interval": "50", 2023-10-20T03:35:10.035 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_log_max": "10000", 2023-10-20T03:35:10.036 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_log_max_summary": "50", 2023-10-20T03:35:10.036 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_log_entries_per_event": "4096", 2023-10-20T03:35:10.036 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_log_epochs": "500", 2023-10-20T03:35:10.036 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_mdsmap_epochs": "500", 2023-10-20T03:35:10.036 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_mgrmap_epochs": "500", 2023-10-20T03:35:10.037 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_osd": "10000", 2023-10-20T03:35:10.037 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_pg_per_osd": "10000", 2023-10-20T03:35:10.037 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_pool_pg_num": "65536", 2023-10-20T03:35:10.037 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_max_snap_prune_per_epoch": "100", 2023-10-20T03:35:10.037 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mds_blocklist_interval": "86400.000000", 2023-10-20T03:35:10.038 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mds_force_trim_to": "0", 2023-10-20T03:35:10.038 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mds_skip_sanity": "false", 2023-10-20T03:35:10.038 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_memory_autotune": "true", 2023-10-20T03:35:10.038 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_memory_target": "2147483648", 2023-10-20T03:35:10.039 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_beacon_grace": "30", 2023-10-20T03:35:10.039 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2023-10-20T03:35:10.039 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_digest_period": "5", 2023-10-20T03:35:10.039 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_inactive_grace": "60", 2023-10-20T03:35:10.039 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_mkfs_grace": "240", 2023-10-20T03:35:10.040 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2023-10-20T03:35:10.040 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_min_osdmap_epochs": "500", 2023-10-20T03:35:10.040 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_complaint_time": "30", 2023-10-20T03:35:10.040 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_history_duration": "600", 2023-10-20T03:35:10.040 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_history_size": "20", 2023-10-20T03:35:10.041 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_history_slow_op_size": "20", 2023-10-20T03:35:10.041 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_history_slow_op_threshold": "10", 2023-10-20T03:35:10.041 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_op_log_threshold": "5", 2023-10-20T03:35:10.041 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_adjust_down_out_interval": "true", 2023-10-20T03:35:10.042 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2023-10-20T03:35:10.042 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2023-10-20T03:35:10.042 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_auto_mark_in": "false", 2023-10-20T03:35:10.042 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_auto_mark_new_in": "true", 2023-10-20T03:35:10.043 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2023-10-20T03:35:10.043 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2023-10-20T03:35:10.043 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_cache_size": "500", 2023-10-20T03:35:10.044 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_cache_size_min": "134217728", 2023-10-20T03:35:10.044 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_crush_smoke_test": "true", 2023-10-20T03:35:10.044 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_destroyed_out_interval": "600", 2023-10-20T03:35:10.044 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_down_out_interval": "600", 2023-10-20T03:35:10.045 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_down_out_subtree_limit": "rack", 2023-10-20T03:35:10.045 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2023-10-20T03:35:10.045 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_force_trim_to": "0", 2023-10-20T03:35:10.045 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_full_ratio": "0.950000", 2023-10-20T03:35:10.046 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2023-10-20T03:35:10.046 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_laggy_halflife": "3600", 2023-10-20T03:35:10.046 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_laggy_max_interval": "300", 2023-10-20T03:35:10.047 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_laggy_weight": "0.300000", 2023-10-20T03:35:10.047 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2023-10-20T03:35:10.048 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_max_creating_pgs": "1024", 2023-10-20T03:35:10.048 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_max_initial_pgs": "1024", 2023-10-20T03:35:10.048 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_min_down_reporters": "2", 2023-10-20T03:35:10.049 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_min_in_ratio": "0.750000", 2023-10-20T03:35:10.049 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_min_up_ratio": "0.300000", 2023-10-20T03:35:10.049 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_nearfull_ratio": "0.850000", 2023-10-20T03:35:10.050 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_prime_pg_temp": "true", 2023-10-20T03:35:10.050 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2023-10-20T03:35:10.050 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2023-10-20T03:35:10.051 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_report_timeout": "900", 2023-10-20T03:35:10.051 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_reporter_subtree_level": "osd", 2023-10-20T03:35:10.051 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2023-10-20T03:35:10.052 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_warn_num_repaired": "10", 2023-10-20T03:35:10.052 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osd_warn_op_age": "32.000000", 2023-10-20T03:35:10.052 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osdmap_full_prune_enabled": "true", 2023-10-20T03:35:10.052 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osdmap_full_prune_interval": "10", 2023-10-20T03:35:10.052 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osdmap_full_prune_min": "10000", 2023-10-20T03:35:10.053 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_osdmap_full_prune_txsize": "100", 2023-10-20T03:35:10.053 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2023-10-20T03:35:10.053 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_stuck_threshold": "60", 2023-10-20T03:35:10.053 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2023-10-20T03:35:10.053 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_warn_min_objects": "10000", 2023-10-20T03:35:10.054 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_warn_min_per_osd": "0", 2023-10-20T03:35:10.054 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pg_warn_min_pool_objects": "1000", 2023-10-20T03:35:10.054 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pool_quota_crit_threshold": "0", 2023-10-20T03:35:10.054 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_pool_quota_warn_threshold": "0", 2023-10-20T03:35:10.055 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_probe_timeout": "2.000000", 2023-10-20T03:35:10.055 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_reweight_max_change": "0.050000", 2023-10-20T03:35:10.055 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_reweight_max_osds": "4", 2023-10-20T03:35:10.055 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_reweight_min_bytes_per_osd": "10", 2023-10-20T03:35:10.056 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_reweight_min_pgs_per_osd": "4", 2023-10-20T03:35:10.056 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2023-10-20T03:35:10.056 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2023-10-20T03:35:10.056 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2023-10-20T03:35:10.056 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_interval": "300", 2023-10-20T03:35:10.057 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_max_keys": "100", 2023-10-20T03:35:10.057 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_scrub_timeout": "300", 2023-10-20T03:35:10.057 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_session_timeout": "300", 2023-10-20T03:35:10.057 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_smart_report_timeout": "5", 2023-10-20T03:35:10.057 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stat_smooth_intervals": "6", 2023-10-20T03:35:10.058 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2023-10-20T03:35:10.058 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stretch_pool_min_size": "2", 2023-10-20T03:35:10.058 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stretch_pool_size": "4", 2023-10-20T03:35:10.058 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2023-10-20T03:35:10.059 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_subscribe_interval": "86400.000000", 2023-10-20T03:35:10.059 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_debug": "false", 2023-10-20T03:35:10.059 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_max_payload_keys": "2000", 2023-10-20T03:35:10.059 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_max_payload_size": "1048576", 2023-10-20T03:35:10.059 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_provider_kill_at": "0", 2023-10-20T03:35:10.060 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_requester_kill_at": "0", 2023-10-20T03:35:10.060 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_sync_timeout": "60.000000", 2023-10-20T03:35:10.060 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_target_pg_per_osd": "100", 2023-10-20T03:35:10.060 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_tick_interval": "5", 2023-10-20T03:35:10.060 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_timecheck_interval": "300.000000", 2023-10-20T03:35:10.061 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_timecheck_skew_interval": "30.000000", 2023-10-20T03:35:10.061 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_use_min_delay_socket": "false", 2023-10-20T03:35:10.061 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_older_version_delay": "604800", 2023-10-20T03:35:10.061 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2023-10-20T03:35:10.061 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2023-10-20T03:35:10.062 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2023-10-20T03:35:10.062 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_filestore_osds": "false", 2023-10-20T03:35:10.062 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2023-10-20T03:35:10.062 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2023-10-20T03:35:10.063 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2023-10-20T03:35:10.063 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_misplaced": "false", 2023-10-20T03:35:10.063 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2023-10-20T03:35:10.063 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_older_version": "true", 2023-10-20T03:35:10.063 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2023-10-20T03:35:10.064 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_pool_no_app": "true", 2023-10-20T03:35:10.064 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_pool_no_redundancy": "false", 2023-10-20T03:35:10.064 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2023-10-20T03:35:10.064 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2023-10-20T03:35:10.064 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2023-10-20T03:35:10.065 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_on_too_few_osds": "false", 2023-10-20T03:35:10.065 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2023-10-20T03:35:10.065 INFO:teuthology.orchestra.run.smithi033.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2023-10-20T03:35:10.065 INFO:teuthology.orchestra.run.smithi033.stdout: "monmap": "", 2023-10-20T03:35:10.066 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-10-20T03:35:10.066 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2023-10-20T03:35:10.066 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2023-10-20T03:35:10.066 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-10-20T03:35:10.066 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2023-10-20T03:35:10.067 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2023-10-20T03:35:10.067 INFO:teuthology.orchestra.run.smithi033.stdout: "motr_tracing_enabled": "false", 2023-10-20T03:35:10.067 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_op_threads": "3", 2023-10-20T03:35:10.067 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_buffer_size": "131072", 2023-10-20T03:35:10.067 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_cm": "false", 2023-10-20T03:35:10.068 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_device_name": "", 2023-10-20T03:35:10.068 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_dscp": "96", 2023-10-20T03:35:10.068 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_enable_hugepage": "false", 2023-10-20T03:35:10.068 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_gid_idx": "0", 2023-10-20T03:35:10.069 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_local_gid": "", 2023-10-20T03:35:10.069 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_polling_us": "1000", 2023-10-20T03:35:10.069 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_port_num": "1", 2023-10-20T03:35:10.069 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_receive_buffers": "32768", 2023-10-20T03:35:10.069 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_receive_queue_len": "4096", 2023-10-20T03:35:10.069 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_roce_ver": "1", 2023-10-20T03:35:10.070 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_send_buffers": "1024", 2023-10-20T03:35:10.070 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_sl": "3", 2023-10-20T03:35:10.070 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_support_srq": "true", 2023-10-20T03:35:10.070 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_rdma_type": "ib", 2023-10-20T03:35:10.071 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_async_reap_threshold": "5", 2023-10-20T03:35:10.071 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_before_connect": "false", 2023-10-20T03:35:10.071 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_ipv4": "true", 2023-10-20T03:35:10.071 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_ipv6": "false", 2023-10-20T03:35:10.072 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_msgr1": "true", 2023-10-20T03:35:10.072 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_msgr2": "true", 2023-10-20T03:35:10.072 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_port_max": "7568", 2023-10-20T03:35:10.072 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_port_min": "6800", 2023-10-20T03:35:10.072 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_prefer_ipv4": "false", 2023-10-20T03:35:10.073 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_retry_count": "3", 2023-10-20T03:35:10.073 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_bind_retry_delay": "5", 2023-10-20T03:35:10.073 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_client": "false", 2023-10-20T03:35:10.073 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_mds": "false", 2023-10-20T03:35:10.073 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_mgr": "false", 2023-10-20T03:35:10.074 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_mon": "false", 2023-10-20T03:35:10.074 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_blackhole_osd": "false", 2023-10-20T03:35:10.074 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_client_mode": "crc secure", 2023-10-20T03:35:10.074 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_cluster_mode": "crc secure", 2023-10-20T03:35:10.075 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_cluster_type": "", 2023-10-20T03:35:10.075 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_compress_secure": "false", 2023-10-20T03:35:10.075 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_connection_idle_timeout": "900", 2023-10-20T03:35:10.075 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_connection_ready_timeout": "10", 2023-10-20T03:35:10.075 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_crc_data": "true", 2023-10-20T03:35:10.076 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_crc_header": "true", 2023-10-20T03:35:10.076 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_bad_msg": "false", 2023-10-20T03:35:10.076 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_bug": "true", 2023-10-20T03:35:10.076 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_old_message": "true", 2023-10-20T03:35:10.077 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_skipped_message": "false", 2023-10-20T03:35:10.077 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_die_on_unhandled_msg": "false", 2023-10-20T03:35:10.077 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dispatch_throttle_bytes": "104857600", 2023-10-20T03:35:10.077 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_coremask": "0xF", 2023-10-20T03:35:10.077 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_debug_allow_loopback": "false", 2023-10-20T03:35:10.078 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_devs_allowlist": "", 2023-10-20T03:35:10.078 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_enable_tso": "true", 2023-10-20T03:35:10.078 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2023-10-20T03:35:10.078 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_host_ipv4_addr": "", 2023-10-20T03:35:10.078 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_hugepages": "", 2023-10-20T03:35:10.079 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_hw_flow_control": "true", 2023-10-20T03:35:10.079 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2023-10-20T03:35:10.079 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_lro": "true", 2023-10-20T03:35:10.079 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_memory_channel": "4", 2023-10-20T03:35:10.080 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2023-10-20T03:35:10.080 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_pmd": "", 2023-10-20T03:35:10.080 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_port_id": "0", 2023-10-20T03:35:10.080 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2023-10-20T03:35:10.080 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dump_corrupt_message_level": "1", 2023-10-20T03:35:10.081 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_dump_on_send": "false", 2023-10-20T03:35:10.081 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_initial_backoff": "0.200000", 2023-10-20T03:35:10.081 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_delay_max": "1.000000", 2023-10-20T03:35:10.081 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_delay_probability": "0.000000", 2023-10-20T03:35:10.082 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_delay_type": "", 2023-10-20T03:35:10.082 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_internal_delays": "0.000000", 2023-10-20T03:35:10.082 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_network_congestion": "0", 2023-10-20T03:35:10.082 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_inject_socket_failures": "2500", 2023-10-20T03:35:10.083 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_learn_addr_from_peer": "true", 2023-10-20T03:35:10.083 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_max_accept_failures": "4", 2023-10-20T03:35:10.083 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_max_backoff": "15.000000", 2023-10-20T03:35:10.083 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_mon_client_mode": "secure crc", 2023-10-20T03:35:10.083 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_mon_cluster_mode": "secure crc", 2023-10-20T03:35:10.084 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_mon_service_mode": "secure crc", 2023-10-20T03:35:10.084 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_osd_compress_min_size": "1024", 2023-10-20T03:35:10.084 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_osd_compress_mode": "none", 2023-10-20T03:35:10.084 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_osd_compression_algorithm": "snappy", 2023-10-20T03:35:10.084 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2023-10-20T03:35:10.085 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_pq_min_cost": "65536", 2023-10-20T03:35:10.085 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_public_type": "", 2023-10-20T03:35:10.086 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_service_mode": "crc secure", 2023-10-20T03:35:10.086 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_tcp_listen_backlog": "512", 2023-10-20T03:35:10.086 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_tcp_nodelay": "true", 2023-10-20T03:35:10.086 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_tcp_prefetch_max_size": "4096", 2023-10-20T03:35:10.087 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_tcp_rcvbuf": "0", 2023-10-20T03:35:10.087 INFO:teuthology.orchestra.run.smithi033.stdout: "ms_type": "async+posix", 2023-10-20T03:35:10.087 INFO:teuthology.orchestra.run.smithi033.stdout: "no_config_file": "false", 2023-10-20T03:35:10.087 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_completion_locks_per_session": "32", 2023-10-20T03:35:10.088 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_debug_inject_relock_delay": "false", 2023-10-20T03:35:10.088 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_inflight_op_bytes": "104857600", 2023-10-20T03:35:10.088 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_inflight_ops": "1024", 2023-10-20T03:35:10.088 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_inject_no_watch_ping": "false", 2023-10-20T03:35:10.088 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_retry_writes_after_first_reply": "false", 2023-10-20T03:35:10.089 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_tick_interval": "5.000000", 2023-10-20T03:35:10.089 INFO:teuthology.orchestra.run.smithi033.stdout: "objecter_timeout": "10.000000", 2023-10-20T03:35:10.089 INFO:teuthology.orchestra.run.smithi033.stdout: "objectstore_blackhole": "false", 2023-10-20T03:35:10.089 INFO:teuthology.orchestra.run.smithi033.stdout: "openssl_engine_opts": "", 2023-10-20T03:35:10.089 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_delay_time": "5.000000", 2023-10-20T03:35:10.089 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_hist_halflife": "1000", 2023-10-20T03:35:10.090 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_max_low_ops": "2", 2023-10-20T03:35:10.090 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_max_ops": "4", 2023-10-20T03:35:10.090 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_min_evict_effort": "0.100000", 2023-10-20T03:35:10.090 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_quantize_effort": "0.100000", 2023-10-20T03:35:10.090 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_agent_slop": "0.020000", 2023-10-20T03:35:10.091 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_aggregated_slow_ops_logging": "true", 2023-10-20T03:35:10.091 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_allow_recovery_below_min_size": "true", 2023-10-20T03:35:10.091 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_asio_thread_count": "2", 2023-10-20T03:35:10.091 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_async_recovery_min_cost": "1", 2023-10-20T03:35:10.091 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_auto_mark_unfound_lost": "false", 2023-10-20T03:35:10.091 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backfill_retry_interval": "30.000000", 2023-10-20T03:35:10.092 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backfill_scan_max": "512", 2023-10-20T03:35:10.092 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backfill_scan_min": "64", 2023-10-20T03:35:10.092 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backoff_on_degraded": "false", 2023-10-20T03:35:10.092 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backoff_on_peering": "false", 2023-10-20T03:35:10.092 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_backoff_on_unfound": "true", 2023-10-20T03:35:10.092 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_beacon_report_interval": "300", 2023-10-20T03:35:10.093 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bench_duration": "30", 2023-10-20T03:35:10.093 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bench_large_size_max_throughput": "104857600", 2023-10-20T03:35:10.093 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bench_max_block_size": "67108864", 2023-10-20T03:35:10.093 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bench_small_size_max_iops": "100", 2023-10-20T03:35:10.093 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_blkin_trace_all": "false", 2023-10-20T03:35:10.094 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_blocked_scrub_grace_period": "120", 2023-10-20T03:35:10.094 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2023-10-20T03:35:10.094 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2023-10-20T03:35:10.094 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2023-10-20T03:35:10.094 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_check_for_log_corruption": "false", 2023-10-20T03:35:10.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_check_max_object_name_len_on_startup": "true", 2023-10-20T03:35:10.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-10-20T03:35:10.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2023-10-20T03:35:10.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-10-20T03:35:10.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_class_update_on_start": "true", 2023-10-20T03:35:10.095 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_client_message_cap": "256", 2023-10-20T03:35:10.096 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_client_message_size_cap": "524288000", 2023-10-20T03:35:10.096 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_client_op_priority": "63", 2023-10-20T03:35:10.096 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_client_watch_timeout": "30", 2023-10-20T03:35:10.096 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_command_max_records": "256", 2023-10-20T03:35:10.096 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_compact_on_start": "false", 2023-10-20T03:35:10.096 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_copyfrom_max_chunk": "8388608", 2023-10-20T03:35:10.097 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_crush_chooseleaf_type": "0", 2023-10-20T03:35:10.097 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_crush_initial_weight": "-1.000000", 2023-10-20T03:35:10.097 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_crush_update_on_start": "true", 2023-10-20T03:35:10.097 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_crush_update_weight_set": "true", 2023-10-20T03:35:10.097 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2023-10-20T03:35:10.098 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2023-10-20T03:35:10.098 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2023-10-20T03:35:10.098 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_drop_ping_duration": "0", 2023-10-20T03:35:10.098 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_drop_ping_probability": "0.000000", 2023-10-20T03:35:10.098 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_feed_pullee": "-1", 2023-10-20T03:35:10.098 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_inject_copyfrom_error": "false", 2023-10-20T03:35:10.099 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2023-10-20T03:35:10.099 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2023-10-20T03:35:10.099 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_misdirected_ops": "false", 2023-10-20T03:35:10.099 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_no_acting_change": "false", 2023-10-20T03:35:10.099 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_no_purge_strays": "false", 2023-10-20T03:35:10.099 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_op_order": "false", 2023-10-20T03:35:10.100 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_pg_log_writeout": "false", 2023-10-20T03:35:10.100 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_pretend_recovery_active": "false", 2023-10-20T03:35:10.100 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_random_push_read_error": "0.000000", 2023-10-20T03:35:10.100 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2023-10-20T03:35:10.100 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_shutdown": "false", 2023-10-20T03:35:10.101 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2023-10-20T03:35:10.101 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2023-10-20T03:35:10.101 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_trim_objects": "false", 2023-10-20T03:35:10.101 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_verify_cached_snaps": "false", 2023-10-20T03:35:10.101 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_verify_missing_on_start": "false", 2023-10-20T03:35:10.101 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_verify_snaps": "false", 2023-10-20T03:35:10.102 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_debug_verify_stray_on_activate": "false", 2023-10-20T03:35:10.102 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_interval": "604800.000000", 2023-10-20T03:35:10.102 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_keys": "1024", 2023-10-20T03:35:10.102 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2023-10-20T03:35:10.102 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2023-10-20T03:35:10.103 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2023-10-20T03:35:10.103 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_stride": "524288", 2023-10-20T03:35:10.103 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2023-10-20T03:35:10.103 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_default_data_pool_replay_window": "5", 2023-10-20T03:35:10.103 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_default_notify_timeout": "30", 2023-10-20T03:35:10.103 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_delete_sleep": "0.000000", 2023-10-20T03:35:10.104 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_delete_sleep_hdd": "5.000000", 2023-10-20T03:35:10.104 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_delete_sleep_hybrid": "1.000000", 2023-10-20T03:35:10.104 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_delete_sleep_ssd": "1.000000", 2023-10-20T03:35:10.104 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_discard_disconnected_ops": "true", 2023-10-20T03:35:10.104 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_enable_op_tracker": "true", 2023-10-20T03:35:10.105 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2023-10-20T03:35:10.105 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_extblkdev_plugins": "vdo", 2023-10-20T03:35:10.105 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_failsafe_full_ratio": "0.970000", 2023-10-20T03:35:10.105 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_fail_on_connection_refused": "true", 2023-10-20T03:35:10.105 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_info": "true", 2023-10-20T03:35:10.105 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_shutdown": "true", 2023-10-20T03:35:10.106 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_shutdown_notify_mon": "true", 2023-10-20T03:35:10.106 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_fast_shutdown_timeout": "15", 2023-10-20T03:35:10.106 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_find_best_info_ignore_history_les": "false", 2023-10-20T03:35:10.106 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_force_auth_primary_missing_objects": "100", 2023-10-20T03:35:10.106 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2023-10-20T03:35:10.106 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_function_tracing": "false", 2023-10-20T03:35:10.107 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_grace": "20", 2023-10-20T03:35:10.107 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_interval": "6", 2023-10-20T03:35:10.107 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2023-10-20T03:35:10.107 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_min_peers": "10", 2023-10-20T03:35:10.107 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_min_size": "2000", 2023-10-20T03:35:10.108 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_stale": "600", 2023-10-20T03:35:10.108 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2023-10-20T03:35:10.108 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_hit_set_max_size": "100000", 2023-10-20T03:35:10.108 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_hit_set_min_size": "1000", 2023-10-20T03:35:10.108 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_hit_set_namespace": ".ceph-internal", 2023-10-20T03:35:10.108 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_ignore_stale_divergent_priors": "false", 2023-10-20T03:35:10.109 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2023-10-20T03:35:10.109 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_inject_failure_on_pg_removal": "false", 2023-10-20T03:35:10.109 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2023-10-20T03:35:10.109 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_journal_flush_on_shutdown": "true", 2023-10-20T03:35:10.109 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_journal_size": "5120", 2023-10-20T03:35:10.109 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_kill_backfill_at": "0", 2023-10-20T03:35:10.110 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_loop_before_reset_tphandle": "64", 2023-10-20T03:35:10.110 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_cache_size": "50", 2023-10-20T03:35:10.110 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_dedup": "true", 2023-10-20T03:35:10.110 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_message_max": "40", 2023-10-20T03:35:10.110 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_message_max_bytes": "10485760", 2023-10-20T03:35:10.111 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_map_share_max_epochs": "40", 2023-10-20T03:35:10.111 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_attr_name_len": "100", 2023-10-20T03:35:10.111 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_attr_size": "0", 2023-10-20T03:35:10.111 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_backfills": "1", 2023-10-20T03:35:10.111 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_markdown_count": "5", 2023-10-20T03:35:10.111 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_markdown_period": "600", 2023-10-20T03:35:10.112 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_object_name_len": "2048", 2023-10-20T03:35:10.112 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_object_namespace_len": "256", 2023-10-20T03:35:10.112 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_object_size": "134217728", 2023-10-20T03:35:10.112 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2023-10-20T03:35:10.112 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_omap_entries_per_request": "1024", 2023-10-20T03:35:10.113 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_pg_blocked_by": "16", 2023-10-20T03:35:10.113 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_pg_log_entries": "10000", 2023-10-20T03:35:10.113 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2023-10-20T03:35:10.113 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_pgls": "1024", 2023-10-20T03:35:10.113 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_push_cost": "8388608", 2023-10-20T03:35:10.113 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_push_objects": "10", 2023-10-20T03:35:10.114 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_scrubs": "3", 2023-10-20T03:35:10.114 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2023-10-20T03:35:10.114 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_trimming_pgs": "2", 2023-10-20T03:35:10.114 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_write_op_reply_len": "64", 2023-10-20T03:35:10.114 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_max_write_size": "90", 2023-10-20T03:35:10.115 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2023-10-20T03:35:10.115 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2023-10-20T03:35:10.115 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2023-10-20T03:35:10.115 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2023-10-20T03:35:10.115 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2023-10-20T03:35:10.115 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2023-10-20T03:35:10.116 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2023-10-20T03:35:10.116 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_override_recovery_settings": "false", 2023-10-20T03:35:10.116 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_profile": "balanced", 2023-10-20T03:35:10.116 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2023-10-20T03:35:10.116 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2023-10-20T03:35:10.117 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2023-10-20T03:35:10.117 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2023-10-20T03:35:10.117 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2023-10-20T03:35:10.117 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2023-10-20T03:35:10.117 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2023-10-20T03:35:10.117 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2023-10-20T03:35:10.118 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2023-10-20T03:35:10.118 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_scheduler_client_wgt": "1", 2023-10-20T03:35:10.118 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mclock_skip_benchmark": "false", 2023-10-20T03:35:10.118 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_base": "805306368", 2023-10-20T03:35:10.118 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_cache_min": "134217728", 2023-10-20T03:35:10.118 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_cache_resize_interval": "1.000000", 2023-10-20T03:35:10.119 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_expected_fragmentation": "0.150000", 2023-10-20T03:35:10.119 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_target": "4294967296", 2023-10-20T03:35:10.119 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_target_autotune": "false", 2023-10-20T03:35:10.119 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2023-10-20T03:35:10.119 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_min_pg_log_entries": "250", 2023-10-20T03:35:10.120 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_min_recovery_priority": "0", 2023-10-20T03:35:10.120 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_heartbeat_interval": "30", 2023-10-20T03:35:10.120 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2023-10-20T03:35:10.120 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_report_interval": "5", 2023-10-20T03:35:10.120 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_report_max_in_flight": "2", 2023-10-20T03:35:10.120 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mon_shutdown_timeout": "5.000000", 2023-10-20T03:35:10.121 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_num_cache_shards": "32", 2023-10-20T03:35:10.121 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_num_op_tracker_shard": "32", 2023-10-20T03:35:10.121 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_numa_auto_affinity": "true", 2023-10-20T03:35:10.121 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_numa_node": "-1", 2023-10-20T03:35:10.121 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_numa_prefer_iface": "true", 2023-10-20T03:35:10.121 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2023-10-20T03:35:10.122 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_objecter_finishers": "1", 2023-10-20T03:35:10.122 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_objectstore": "bluestore", 2023-10-20T03:35:10.122 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_objectstore_fuse": "false", 2023-10-20T03:35:10.122 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_objectstore_tracing": "false", 2023-10-20T03:35:10.122 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_complaint_time": "30.000000", 2023-10-20T03:35:10.122 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_history_duration": "600", 2023-10-20T03:35:10.123 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_history_size": "20", 2023-10-20T03:35:10.123 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_history_slow_op_size": "20", 2023-10-20T03:35:10.123 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2023-10-20T03:35:10.123 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_log_threshold": "5", 2023-10-20T03:35:10.123 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_shards": "0", 2023-10-20T03:35:10.124 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_shards_hdd": "5", 2023-10-20T03:35:10.124 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_shards_ssd": "8", 2023-10-20T03:35:10.124 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_threads_per_shard": "0", 2023-10-20T03:35:10.124 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2023-10-20T03:35:10.124 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2023-10-20T03:35:10.125 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2023-10-20T03:35:10.125 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_pq_min_cost": "65536", 2023-10-20T03:35:10.125 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_queue": "mclock_scheduler", 2023-10-20T03:35:10.125 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_queue_cut_off": "high", 2023-10-20T03:35:10.125 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_thread_suicide_timeout": "150", 2023-10-20T03:35:10.125 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_op_thread_timeout": "15", 2023-10-20T03:35:10.126 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_open_classes_on_start": "true", 2023-10-20T03:35:10.126 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_os_flags": "0", 2023-10-20T03:35:10.126 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_peering_op_priority": "255", 2023-10-20T03:35:10.126 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_delete_cost": "1048576", 2023-10-20T03:35:10.126 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_delete_priority": "5", 2023-10-20T03:35:10.126 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2023-10-20T03:35:10.127 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2023-10-20T03:35:10.127 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_log_dups_tracked": "3000", 2023-10-20T03:35:10.127 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_log_trim_max": "10000", 2023-10-20T03:35:10.127 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_log_trim_min": "100", 2023-10-20T03:35:10.127 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2023-10-20T03:35:10.128 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_object_context_cache_count": "64", 2023-10-20T03:35:10.128 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pg_stat_report_interval_max": "500", 2023-10-20T03:35:10.128 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2023-10-20T03:35:10.128 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_min_evict_age": "0", 2023-10-20T03:35:10.128 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_min_flush_age": "0", 2023-10-20T03:35:10.128 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2023-10-20T03:35:10.129 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2023-10-20T03:35:10.129 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2023-10-20T03:35:10.129 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_crimson": "false", 2023-10-20T03:35:10.129 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_crush_rule": "-1", 2023-10-20T03:35:10.129 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_ec_fast_read": "false", 2023-10-20T03:35:10.130 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2023-10-20T03:35:10.130 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_bulk": "false", 2023-10-20T03:35:10.130 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_hashpspool": "true", 2023-10-20T03:35:10.130 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_nodelete": "false", 2023-10-20T03:35:10.130 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_nopgchange": "false", 2023-10-20T03:35:10.130 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flag_nosizechange": "false", 2023-10-20T03:35:10.131 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_flags": "0", 2023-10-20T03:35:10.131 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2023-10-20T03:35:10.131 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_min_size": "0", 2023-10-20T03:35:10.131 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2023-10-20T03:35:10.131 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_pg_num": "32", 2023-10-20T03:35:10.131 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_pgp_num": "0", 2023-10-20T03:35:10.132 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2023-10-20T03:35:10.132 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_size": "2", 2023-10-20T03:35:10.132 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_default_type": "replicated", 2023-10-20T03:35:10.132 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2023-10-20T03:35:10.132 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pool_use_gmt_hitset": "true", 2023-10-20T03:35:10.132 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_push_per_object_cost": "1000", 2023-10-20T03:35:10.133 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_read_ec_check_for_errors": "false", 2023-10-20T03:35:10.133 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recover_clone_overlap": "true", 2023-10-20T03:35:10.133 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recover_clone_overlap_limit": "10", 2023-10-20T03:35:10.133 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_cost": "20971520", 2023-10-20T03:35:10.133 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_delay_start": "0.000000", 2023-10-20T03:35:10.134 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_active": "0", 2023-10-20T03:35:10.134 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_active_hdd": "3", 2023-10-20T03:35:10.134 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_active_ssd": "10", 2023-10-20T03:35:10.134 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_chunk": "8388608", 2023-10-20T03:35:10.134 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2023-10-20T03:35:10.134 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_max_single_start": "1", 2023-10-20T03:35:10.135 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_op_priority": "3", 2023-10-20T03:35:10.135 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_op_warn_multiple": "16", 2023-10-20T03:35:10.135 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_priority": "5", 2023-10-20T03:35:10.135 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_retry_interval": "30.000000", 2023-10-20T03:35:10.135 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_sleep": "0.000000", 2023-10-20T03:35:10.135 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_sleep_hdd": "0.100000", 2023-10-20T03:35:10.136 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2023-10-20T03:35:10.136 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_recovery_sleep_ssd": "0.000000", 2023-10-20T03:35:10.136 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_repair_during_recovery": "false", 2023-10-20T03:35:10.136 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_requested_scrub_priority": "120", 2023-10-20T03:35:10.136 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2023-10-20T03:35:10.137 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_rollback_to_cluster_snap": "", 2023-10-20T03:35:10.137 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_auto_repair": "false", 2023-10-20T03:35:10.137 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_auto_repair_num_errors": "5", 2023-10-20T03:35:10.137 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_backoff_ratio": "0.660000", 2023-10-20T03:35:10.137 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_begin_hour": "0", 2023-10-20T03:35:10.137 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_begin_week_day": "0", 2023-10-20T03:35:10.138 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_chunk_max": "25", 2023-10-20T03:35:10.138 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_chunk_min": "5", 2023-10-20T03:35:10.138 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_cost": "52428800", 2023-10-20T03:35:10.138 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_during_recovery": "false", 2023-10-20T03:35:10.138 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_end_hour": "0", 2023-10-20T03:35:10.139 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_end_week_day": "0", 2023-10-20T03:35:10.139 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_event_cost": "4096", 2023-10-20T03:35:10.139 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_extended_sleep": "0.000000", 2023-10-20T03:35:10.139 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2023-10-20T03:35:10.139 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_invalid_stats": "true", 2023-10-20T03:35:10.139 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_load_threshold": "0.500000", 2023-10-20T03:35:10.140 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_max_interval": "604800.000000", 2023-10-20T03:35:10.140 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_max_preemptions": "5", 2023-10-20T03:35:10.140 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_min_interval": "86400.000000", 2023-10-20T03:35:10.140 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_priority": "5", 2023-10-20T03:35:10.140 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_reservation_timeout": "5000", 2023-10-20T03:35:10.141 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_sleep": "0.000000", 2023-10-20T03:35:10.141 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_scrub_slow_reservation_response": "2200", 2023-10-20T03:35:10.141 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_shallow_scrub_chunk_max": "100", 2023-10-20T03:35:10.141 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_shallow_scrub_chunk_min": "50", 2023-10-20T03:35:10.141 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_shutdown_pgref_assert": "false", 2023-10-20T03:35:10.141 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_skip_data_digest": "false", 2023-10-20T03:35:10.142 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_smart_report_timeout": "5", 2023-10-20T03:35:10.142 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_cost": "1048576", 2023-10-20T03:35:10.142 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_priority": "5", 2023-10-20T03:35:10.142 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_sleep": "0.000000", 2023-10-20T03:35:10.142 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2023-10-20T03:35:10.142 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2023-10-20T03:35:10.143 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2023-10-20T03:35:10.143 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_stats_update_period_not_scrubbing": "120", 2023-10-20T03:35:10.143 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_stats_update_period_scrubbing": "15", 2023-10-20T03:35:10.143 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2023-10-20T03:35:10.143 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_target_transaction_size": "30", 2023-10-20T03:35:10.144 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_count": "4", 2023-10-20T03:35:10.144 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2023-10-20T03:35:10.144 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2023-10-20T03:35:10.144 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2023-10-20T03:35:10.144 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2023-10-20T03:35:10.145 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2023-10-20T03:35:10.145 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2023-10-20T03:35:10.145 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_default_cache_mode": "writeback", 2023-10-20T03:35:10.145 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2023-10-20T03:35:10.145 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tier_promote_max_objects_sec": "25", 2023-10-20T03:35:10.145 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_tracing": "false", 2023-10-20T03:35:10.146 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_use_stale_snap": "false", 2023-10-20T03:35:10.146 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2023-10-20T03:35:10.146 INFO:teuthology.orchestra.run.smithi033.stdout: "osdc_blkin_trace_all": "false", 2023-10-20T03:35:10.146 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_kill_at": "0", 2023-10-20T03:35:10.146 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_max_join_drift": "10", 2023-10-20T03:35:10.146 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_min": "500", 2023-10-20T03:35:10.147 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_min_wait": "0.050000", 2023-10-20T03:35:10.147 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_propose_interval": "1.000000", 2023-10-20T03:35:10.147 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_service_trim_max": "500", 2023-10-20T03:35:10.147 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_service_trim_max_multiplier": "20", 2023-10-20T03:35:10.147 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_service_trim_min": "250", 2023-10-20T03:35:10.148 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_stash_full_interval": "25", 2023-10-20T03:35:10.148 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_trim_max": "500", 2023-10-20T03:35:10.148 INFO:teuthology.orchestra.run.smithi033.stdout: "paxos_trim_min": "250", 2023-10-20T03:35:10.148 INFO:teuthology.orchestra.run.smithi033.stdout: "perf": "true", 2023-10-20T03:35:10.148 INFO:teuthology.orchestra.run.smithi033.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2023-10-20T03:35:10.148 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin_crypto_accelerator": "crypto_isal", 2023-10-20T03:35:10.149 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin_dir": "/usr/lib/ceph", 2023-10-20T03:35:10.149 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "-", 2023-10-20T03:35:10.149 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrv": "", 2023-10-20T03:35:10.149 INFO:teuthology.orchestra.run.smithi033.stdout: "public_bind_addr": "-", 2023-10-20T03:35:10.149 INFO:teuthology.orchestra.run.smithi033.stdout: "public_network": "", 2023-10-20T03:35:10.150 INFO:teuthology.orchestra.run.smithi033.stdout: "public_network_interface": "", 2023-10-20T03:35:10.150 INFO:teuthology.orchestra.run.smithi033.stdout: "qat_compressor_enabled": "false", 2023-10-20T03:35:10.150 INFO:teuthology.orchestra.run.smithi033.stdout: "qat_compressor_session_max_number": "256", 2023-10-20T03:35:10.150 INFO:teuthology.orchestra.run.smithi033.stdout: "rados_mon_op_timeout": "0", 2023-10-20T03:35:10.150 INFO:teuthology.orchestra.run.smithi033.stdout: "rados_osd_op_timeout": "0", 2023-10-20T03:35:10.150 INFO:teuthology.orchestra.run.smithi033.stdout: "rados_tracing": "false", 2023-10-20T03:35:10.151 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_atime_update_interval": "60", 2023-10-20T03:35:10.151 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2023-10-20T03:35:10.151 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_balance_parent_reads": "false", 2023-10-20T03:35:10.151 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_balance_snap_reads": "false", 2023-10-20T03:35:10.151 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_blkin_trace_all": "false", 2023-10-20T03:35:10.151 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_blocklist_expire_seconds": "0", 2023-10-20T03:35:10.152 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_blocklist_on_break_lock": "true", 2023-10-20T03:35:10.152 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache": "true", 2023-10-20T03:35:10.152 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_block_writes_upfront": "false", 2023-10-20T03:35:10.152 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_max_dirty": "25165824", 2023-10-20T03:35:10.152 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_max_dirty_age": "1.000000", 2023-10-20T03:35:10.153 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_max_dirty_object": "0", 2023-10-20T03:35:10.153 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_policy": "writearound", 2023-10-20T03:35:10.153 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_size": "33554432", 2023-10-20T03:35:10.153 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_target_dirty": "16777216", 2023-10-20T03:35:10.153 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_cache_writethrough_until_flush": "true", 2023-10-20T03:35:10.153 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_clone_copy_on_read": "false", 2023-10-20T03:35:10.154 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_compression_hint": "none", 2023-10-20T03:35:10.154 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_concurrent_management_ops": "10", 2023-10-20T03:35:10.154 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_config_pool_override_update_timestamp": "0", 2023-10-20T03:35:10.154 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_clone_format": "auto", 2023-10-20T03:35:10.154 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_data_pool": "", 2023-10-20T03:35:10.155 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_features": "61", 2023-10-20T03:35:10.155 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_format": "2", 2023-10-20T03:35:10.155 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_map_options": "", 2023-10-20T03:35:10.155 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_order": "22", 2023-10-20T03:35:10.155 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_pool": "rbd", 2023-10-20T03:35:10.155 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2023-10-20T03:35:10.156 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_stripe_count": "0", 2023-10-20T03:35:10.156 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_default_stripe_unit": "0", 2023-10-20T03:35:10.156 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_disable_zero_copy_writes": "true", 2023-10-20T03:35:10.156 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_discard_granularity_bytes": "65536", 2023-10-20T03:35:10.156 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_discard_on_zeroed_write_same": "true", 2023-10-20T03:35:10.156 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_enable_alloc_hint": "true", 2023-10-20T03:35:10.157 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2023-10-20T03:35:10.157 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_io_scheduler": "simple", 2023-10-20T03:35:10.157 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2023-10-20T03:35:10.157 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_commit_age": "5.000000", 2023-10-20T03:35:10.157 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2023-10-20T03:35:10.157 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_max_payload_bytes": "16384", 2023-10-20T03:35:10.158 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_flush_age": "0.000000", 2023-10-20T03:35:10.158 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_flush_bytes": "1048576", 2023-10-20T03:35:10.158 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_flush_interval": "0", 2023-10-20T03:35:10.158 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2023-10-20T03:35:10.158 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2023-10-20T03:35:10.159 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_order": "24", 2023-10-20T03:35:10.159 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_pool": "", 2023-10-20T03:35:10.159 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_journal_splay_width": "4", 2023-10-20T03:35:10.159 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_localize_parent_reads": "false", 2023-10-20T03:35:10.159 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_localize_snap_reads": "false", 2023-10-20T03:35:10.159 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2023-10-20T03:35:10.160 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2023-10-20T03:35:10.160 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2023-10-20T03:35:10.160 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2023-10-20T03:35:10.160 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2023-10-20T03:35:10.160 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2023-10-20T03:35:10.161 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_policy_type": "simple", 2023-10-20T03:35:10.161 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2023-10-20T03:35:10.161 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_image_state_check_interval": "30", 2023-10-20T03:35:10.161 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2023-10-20T03:35:10.161 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2023-10-20T03:35:10.161 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2023-10-20T03:35:10.162 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2023-10-20T03:35:10.162 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2023-10-20T03:35:10.162 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_autotune": "true", 2023-10-20T03:35:10.162 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_base": "805306368", 2023-10-20T03:35:10.162 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2023-10-20T03:35:10.163 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_cache_min": "134217728", 2023-10-20T03:35:10.163 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2023-10-20T03:35:10.163 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2023-10-20T03:35:10.163 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_memory_target": "4294967296", 2023-10-20T03:35:10.163 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_perf_stats_prio": "5", 2023-10-20T03:35:10.163 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2023-10-20T03:35:10.164 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2023-10-20T03:35:10.164 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirroring_delete_delay": "0", 2023-10-20T03:35:10.164 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2023-10-20T03:35:10.164 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirroring_replay_delay": "0", 2023-10-20T03:35:10.164 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2023-10-20T03:35:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2023-10-20T03:35:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_move_to_trash_on_remove": "false", 2023-10-20T03:35:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2023-10-20T03:35:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_mtime_update_interval": "60", 2023-10-20T03:35:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_non_blocking_aio": "true", 2023-10-20T03:35:10.165 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_op_thread_timeout": "60", 2023-10-20T03:35:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_op_threads": "1", 2023-10-20T03:35:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_parent_cache_enabled": "false", 2023-10-20T03:35:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_persistent_cache_mode": "disabled", 2023-10-20T03:35:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_persistent_cache_path": "/tmp", 2023-10-20T03:35:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_persistent_cache_size": "1073741824", 2023-10-20T03:35:10.166 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_plugins": "", 2023-10-20T03:35:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_bps_burst": "0", 2023-10-20T03:35:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_bps_burst_seconds": "1", 2023-10-20T03:35:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_bps_limit": "0", 2023-10-20T03:35:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_exclude_ops": "0", 2023-10-20T03:35:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_iops_burst": "0", 2023-10-20T03:35:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_iops_burst_seconds": "1", 2023-10-20T03:35:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_iops_limit": "0", 2023-10-20T03:35:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_bps_burst": "0", 2023-10-20T03:35:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2023-10-20T03:35:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_bps_limit": "0", 2023-10-20T03:35:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_iops_burst": "0", 2023-10-20T03:35:10.168 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2023-10-20T03:35:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_read_iops_limit": "0", 2023-10-20T03:35:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_schedule_tick_min": "50", 2023-10-20T03:35:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_bps_burst": "0", 2023-10-20T03:35:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2023-10-20T03:35:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_bps_limit": "0", 2023-10-20T03:35:10.169 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_iops_burst": "0", 2023-10-20T03:35:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2023-10-20T03:35:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_qos_write_iops_limit": "0", 2023-10-20T03:35:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_quiesce_notification_attempts": "10", 2023-10-20T03:35:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_read_from_replica_policy": "default", 2023-10-20T03:35:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2023-10-20T03:35:10.170 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_readahead_max_bytes": "524288", 2023-10-20T03:35:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_readahead_trigger_requests": "10", 2023-10-20T03:35:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_request_timed_out_seconds": "30", 2023-10-20T03:35:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_skip_partial_discard": "true", 2023-10-20T03:35:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2023-10-20T03:35:10.171 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_tracing": "false", 2023-10-20T03:35:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_validate_names": "true", 2023-10-20T03:35:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_validate_pool": "true", 2023-10-20T03:35:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "restapi_base_url": "", 2023-10-20T03:35:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "restapi_log_level": "", 2023-10-20T03:35:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_acl_grants_max_num": "100", 2023-10-20T03:35:10.172 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_admin_entry": "admin", 2023-10-20T03:35:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2023-10-20T03:35:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_backend_store": "rados", 2023-10-20T03:35:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_barbican_url": "", 2023-10-20T03:35:10.173 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_beast_enable_async": "true", 2023-10-20T03:35:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2023-10-20T03:35:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_default_quota_max_size": "-1", 2023-10-20T03:35:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_index_max_aio": "128", 2023-10-20T03:35:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2023-10-20T03:35:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_quota_cache_size": "10000", 2023-10-20T03:35:10.174 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_quota_ttl": "600", 2023-10-20T03:35:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_bucket_sync_spawn_window": "20", 2023-10-20T03:35:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cache_enabled": "true", 2023-10-20T03:35:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cache_expiry_interval": "900", 2023-10-20T03:35:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cache_lru_size": "10000", 2023-10-20T03:35:10.175 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_config_store": "rados", 2023-10-20T03:35:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_content_length_compat": "false", 2023-10-20T03:35:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_copy_obj_progress": "true", 2023-10-20T03:35:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2023-10-20T03:35:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cors_rules_max_num": "100", 2023-10-20T03:35:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_cross_domain_policy": "", 2023-10-20T03:35:10.176 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_default_encryption_key": "", 2023-10-20T03:35:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_addr": "", 2023-10-20T03:35:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_ca_path": "", 2023-10-20T03:35:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_client_cert": "", 2023-10-20T03:35:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_client_key": "", 2023-10-20T03:35:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_kms_key_template": "", 2023-10-20T03:35:10.177 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_password": "", 2023-10-20T03:35:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2023-10-20T03:35:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_kmip_username": "", 2023-10-20T03:35:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_require_ssl": "true", 2023-10-20T03:35:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2023-10-20T03:35:10.178 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2023-10-20T03:35:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_backend": "vault", 2023-10-20T03:35:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2023-10-20T03:35:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2023-10-20T03:35:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2023-10-20T03:35:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2023-10-20T03:35:10.179 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2023-10-20T03:35:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2023-10-20T03:35:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2023-10-20T03:35:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2023-10-20T03:35:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2023-10-20T03:35:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2023-10-20T03:35:10.180 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2023-10-20T03:35:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_suppress_logs": "true", 2023-10-20T03:35:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_addr": "", 2023-10-20T03:35:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_auth": "token", 2023-10-20T03:35:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_namespace": "", 2023-10-20T03:35:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_prefix": "", 2023-10-20T03:35:10.181 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_secret_engine": "transit", 2023-10-20T03:35:10.182 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_ssl_cacert": "", 2023-10-20T03:35:10.182 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2023-10-20T03:35:10.182 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2023-10-20T03:35:10.182 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_token_file": "", 2023-10-20T03:35:10.182 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_crypt_vault_verify_ssl": "true", 2023-10-20T03:35:10.183 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_buffersize": "524288", 2023-10-20T03:35:10.183 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_low_speed_limit": "1024", 2023-10-20T03:35:10.183 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_low_speed_time": "300", 2023-10-20T03:35:10.183 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_tcp_keepalive": "0", 2023-10-20T03:35:10.183 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_curl_wait_timeout_ms": "1000", 2023-10-20T03:35:10.183 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2023-10-20T03:35:10.184 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2023-10-20T03:35:10.184 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2023-10-20T03:35:10.184 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2023-10-20T03:35:10.184 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_l1_fadvise": "4", 2023-10-20T03:35:10.184 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2023-10-20T03:35:10.185 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_libaio_aio_num": "64", 2023-10-20T03:35:10.185 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d3n_libaio_aio_threads": "20", 2023-10-20T03:35:10.185 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d4n_host": "127.0.0.1", 2023-10-20T03:35:10.185 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_d4n_port": "6379", 2023-10-20T03:35:10.185 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2023-10-20T03:35:10.185 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_log_changes_size": "1000", 2023-10-20T03:35:10.186 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_log_num_shards": "128", 2023-10-20T03:35:10.186 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_log_obj_prefix": "data_log", 2023-10-20T03:35:10.186 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_log_window": "30", 2023-10-20T03:35:10.186 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_notify_interval_msec": "0", 2023-10-20T03:35:10.186 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_sync_poll_interval": "20", 2023-10-20T03:35:10.186 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_data_sync_spawn_window": "20", 2023-10-20T03:35:10.187 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2023-10-20T03:35:10.187 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_debug_inject_set_olh_err": "0", 2023-10-20T03:35:10.187 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_data_log_backing": "fifo", 2023-10-20T03:35:10.187 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_realm_info_oid": "default.realm", 2023-10-20T03:35:10.187 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_region_info_oid": "default.region", 2023-10-20T03:35:10.188 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_zone_info_oid": "default.zone", 2023-10-20T03:35:10.188 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2023-10-20T03:35:10.188 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_defer_to_bucket_acls": "", 2023-10-20T03:35:10.188 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_delete_multi_obj_max_num": "1000", 2023-10-20T03:35:10.188 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_admin_lim": "0.000000", 2023-10-20T03:35:10.189 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_admin_res": "100.000000", 2023-10-20T03:35:10.189 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2023-10-20T03:35:10.189 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_auth_lim": "0.000000", 2023-10-20T03:35:10.189 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_auth_res": "200.000000", 2023-10-20T03:35:10.189 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2023-10-20T03:35:10.189 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_data_lim": "0.000000", 2023-10-20T03:35:10.190 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_data_res": "500.000000", 2023-10-20T03:35:10.190 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_data_wgt": "500.000000", 2023-10-20T03:35:10.190 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2023-10-20T03:35:10.190 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_metadata_res": "500.000000", 2023-10-20T03:35:10.190 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2023-10-20T03:35:10.190 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dns_name": "", 2023-10-20T03:35:10.191 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dns_s3website_name": "", 2023-10-20T03:35:10.191 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_dynamic_resharding": "true", 2023-10-20T03:35:10.191 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2023-10-20T03:35:10.191 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_gc_threads": "true", 2023-10-20T03:35:10.191 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_lc_threads": "true", 2023-10-20T03:35:10.192 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_ops_log": "false", 2023-10-20T03:35:10.192 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_quota_threads": "true", 2023-10-20T03:35:10.192 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_static_website": "false", 2023-10-20T03:35:10.192 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enable_usage_log": "false", 2023-10-20T03:35:10.192 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_enforce_swift_acls": "true", 2023-10-20T03:35:10.192 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_exit_timeout_secs": "120", 2023-10-20T03:35:10.193 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_expose_bucket": "false", 2023-10-20T03:35:10.193 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_extended_http_attrs": "", 2023-10-20T03:35:10.193 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_filter": "none", 2023-10-20T03:35:10.193 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2023-10-20T03:35:10.193 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_frontends": "beast port=7480", 2023-10-20T03:35:10.194 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_concurrent_io": "10", 2023-10-20T03:35:10.194 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_deferred": "50", 2023-10-20T03:35:10.194 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2023-10-20T03:35:10.194 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_objs": "32", 2023-10-20T03:35:10.194 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_queue_size": "134213632", 2023-10-20T03:35:10.194 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_max_trim_chunk": "16", 2023-10-20T03:35:10.195 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_obj_min_wait": "7200", 2023-10-20T03:35:10.195 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_processor_max_time": "3600", 2023-10-20T03:35:10.195 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_gc_processor_period": "3600", 2023-10-20T03:35:10.195 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_get_obj_max_req_size": "4194304", 2023-10-20T03:35:10.195 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_get_obj_window_size": "16777216", 2023-10-20T03:35:10.196 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_healthcheck_disabling_path": "", 2023-10-20T03:35:10.196 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ignore_get_invalid_range": "false", 2023-10-20T03:35:10.196 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_init_timeout": "300", 2023-10-20T03:35:10.196 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2023-10-20T03:35:10.196 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2023-10-20T03:35:10.196 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_accepted_admin_roles": "", 2023-10-20T03:35:10.197 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_accepted_reader_roles": "", 2023-10-20T03:35:10.197 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2023-10-20T03:35:10.197 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_domain": "", 2023-10-20T03:35:10.197 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_password": "", 2023-10-20T03:35:10.197 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_password_path": "", 2023-10-20T03:35:10.197 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_project": "", 2023-10-20T03:35:10.198 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_tenant": "", 2023-10-20T03:35:10.198 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_token": "", 2023-10-20T03:35:10.198 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_token_path": "", 2023-10-20T03:35:10.198 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_admin_user": "", 2023-10-20T03:35:10.198 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_api_version": "2", 2023-10-20T03:35:10.199 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_domain": "", 2023-10-20T03:35:10.199 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_password": "", 2023-10-20T03:35:10.199 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_project": "", 2023-10-20T03:35:10.199 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_tenant": "", 2023-10-20T03:35:10.199 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_barbican_user": "", 2023-10-20T03:35:10.199 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2023-10-20T03:35:10.200 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_implicit_tenants": "false", 2023-10-20T03:35:10.200 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2023-10-20T03:35:10.200 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_service_token_enabled": "false", 2023-10-20T03:35:10.200 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_token_cache_size": "10000", 2023-10-20T03:35:10.200 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_url": "", 2023-10-20T03:35:10.200 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_keystone_verify_ssl": "true", 2023-10-20T03:35:10.201 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_debug_interval": "-1", 2023-10-20T03:35:10.201 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_lock_max_time": "90", 2023-10-20T03:35:10.201 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_max_objs": "32", 2023-10-20T03:35:10.201 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_max_rules": "1000", 2023-10-20T03:35:10.201 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_max_worker": "3", 2023-10-20T03:35:10.201 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_max_wp_worker": "3", 2023-10-20T03:35:10.202 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lc_thread_delay": "0", 2023-10-20T03:35:10.202 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2023-10-20T03:35:10.202 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_dnattr": "uid", 2023-10-20T03:35:10.202 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2023-10-20T03:35:10.202 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_searchfilter": "", 2023-10-20T03:35:10.202 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2023-10-20T03:35:10.203 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ldap_uri": "ldaps://", 2023-10-20T03:35:10.203 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2023-10-20T03:35:10.203 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_list_bucket_min_readahead": "1000", 2023-10-20T03:35:10.203 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_list_buckets_max_chunk": "1000", 2023-10-20T03:35:10.203 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_log_http_headers": "", 2023-10-20T03:35:10.203 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_log_nonexistent_bucket": "false", 2023-10-20T03:35:10.204 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2023-10-20T03:35:10.204 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_log_object_name_utc": "false", 2023-10-20T03:35:10.204 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_lua_max_memory_per_state": "128000", 2023-10-20T03:35:10.204 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2023-10-20T03:35:10.204 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_attr_name_len": "0", 2023-10-20T03:35:10.205 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_attr_size": "0", 2023-10-20T03:35:10.205 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_attrs_num_in_req": "0", 2023-10-20T03:35:10.205 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_chunk_size": "4194304", 2023-10-20T03:35:10.205 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_concurrent_requests": "1024", 2023-10-20T03:35:10.205 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2023-10-20T03:35:10.205 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_dynamic_shards": "1999", 2023-10-20T03:35:10.206 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_listing_results": "1000", 2023-10-20T03:35:10.206 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_notify_retries": "10", 2023-10-20T03:35:10.206 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_objs_per_shard": "100000", 2023-10-20T03:35:10.206 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_put_param_size": "1048576", 2023-10-20T03:35:10.206 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_put_size": "5368709120", 2023-10-20T03:35:10.206 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_max_slo_entries": "1000", 2023-10-20T03:35:10.207 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_md_log_max_shards": "64", 2023-10-20T03:35:10.207 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_md_notify_interval_msec": "200", 2023-10-20T03:35:10.207 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_meta_sync_poll_interval": "20", 2023-10-20T03:35:10.207 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_meta_sync_spawn_window": "20", 2023-10-20T03:35:10.207 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_mime_types_file": "/etc/mime.types", 2023-10-20T03:35:10.208 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_mp_lock_max_time": "600", 2023-10-20T03:35:10.208 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_multi_obj_del_max_aio": "16", 2023-10-20T03:35:10.208 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_multipart_min_part_size": "5242880", 2023-10-20T03:35:10.208 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_multipart_part_upload_limit": "10000", 2023-10-20T03:35:10.208 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_fhcache_partitions": "3", 2023-10-20T03:35:10.208 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_fhcache_size": "2017", 2023-10-20T03:35:10.209 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_frontends": "rgw-nfs", 2023-10-20T03:35:10.209 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2023-10-20T03:35:10.209 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_lru_lanes": "5", 2023-10-20T03:35:10.209 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_max_gc": "300", 2023-10-20T03:35:10.209 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_namespace_expire_secs": "300", 2023-10-20T03:35:10.209 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_run_gc_threads": "false", 2023-10-20T03:35:10.210 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_run_lc_threads": "false", 2023-10-20T03:35:10.210 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_run_quota_threads": "false", 2023-10-20T03:35:10.210 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_run_sync_thread": "false", 2023-10-20T03:35:10.210 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_s3_fast_attrs": "false", 2023-10-20T03:35:10.210 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_nfs_write_completion_interval_s": "10", 2023-10-20T03:35:10.211 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_num_async_rados_threads": "32", 2023-10-20T03:35:10.211 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_num_control_oids": "8", 2023-10-20T03:35:10.211 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_numa_node": "-1", 2023-10-20T03:35:10.211 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_obj_stripe_size": "4194304", 2023-10-20T03:35:10.211 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_obj_tombstone_cache_size": "1000", 2023-10-20T03:35:10.211 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_objexp_chunk_size": "100", 2023-10-20T03:35:10.212 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_objexp_gc_interval": "600", 2023-10-20T03:35:10.212 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_objexp_hints_num_shards": "127", 2023-10-20T03:35:10.212 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_olh_pending_timeout_sec": "3600", 2023-10-20T03:35:10.212 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_op_thread_suicide_timeout": "0", 2023-10-20T03:35:10.212 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_op_thread_timeout": "600", 2023-10-20T03:35:10.213 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_op_tracing": "false", 2023-10-20T03:35:10.213 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_opa_token": "", 2023-10-20T03:35:10.213 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_opa_url": "", 2023-10-20T03:35:10.213 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_opa_verify_ssl": "true", 2023-10-20T03:35:10.213 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ops_log_data_backlog": "5242880", 2023-10-20T03:35:10.213 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2023-10-20T03:35:10.214 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ops_log_rados": "false", 2023-10-20T03:35:10.214 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_ops_log_socket_path": "", 2023-10-20T03:35:10.214 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_override_bucket_index_max_shards": "0", 2023-10-20T03:35:10.214 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2023-10-20T03:35:10.214 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2023-10-20T03:35:10.215 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_period_push_interval": "2.000000", 2023-10-20T03:35:10.215 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_period_push_interval_max": "30.000000", 2023-10-20T03:35:10.215 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_period_root_pool": ".rgw.root", 2023-10-20T03:35:10.215 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_policy_reject_invalid_principals": "true", 2023-10-20T03:35:10.215 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2023-10-20T03:35:10.215 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_posix_cache_lanes": "3", 2023-10-20T03:35:10.216 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_posix_cache_lmdb_count": "3", 2023-10-20T03:35:10.216 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_posix_cache_max_buckets": "100", 2023-10-20T03:35:10.216 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_posix_cache_partitions": "3", 2023-10-20T03:35:10.216 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2023-10-20T03:35:10.216 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_print_continue": "true", 2023-10-20T03:35:10.216 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_print_prohibited_content_length": "false", 2023-10-20T03:35:10.217 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_put_obj_max_window_size": "67108864", 2023-10-20T03:35:10.217 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_put_obj_min_window_size": "16777216", 2023-10-20T03:35:10.217 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2023-10-20T03:35:10.217 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_rados_pool_recovery_priority": "5", 2023-10-20T03:35:10.217 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_rados_tracing": "false", 2023-10-20T03:35:10.217 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_realm": "", 2023-10-20T03:35:10.218 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_realm_id": "", 2023-10-20T03:35:10.218 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_realm_root_pool": ".rgw.root", 2023-10-20T03:35:10.218 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_region": "", 2023-10-20T03:35:10.218 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_region_root_pool": ".rgw.root", 2023-10-20T03:35:10.218 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_relaxed_region_enforcement": "false", 2023-10-20T03:35:10.219 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_relaxed_s3_bucket_names": "false", 2023-10-20T03:35:10.219 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2023-10-20T03:35:10.219 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_request_uri": "", 2023-10-20T03:35:10.219 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_batch_size": "64", 2023-10-20T03:35:10.219 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_bucket_lock_duration": "360", 2023-10-20T03:35:10.219 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_max_aio": "128", 2023-10-20T03:35:10.220 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_num_logs": "16", 2023-10-20T03:35:10.220 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_reshard_thread_interval": "600", 2023-10-20T03:35:10.220 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_resolve_cname": "false", 2023-10-20T03:35:10.220 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_rest_getusage_op_compat": "false", 2023-10-20T03:35:10.220 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_run_sync_thread": "true", 2023-10-20T03:35:10.221 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_order": "sts, external, local", 2023-10-20T03:35:10.221 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_use_keystone": "false", 2023-10-20T03:35:10.221 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_use_ldap": "false", 2023-10-20T03:35:10.221 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_use_rados": "true", 2023-10-20T03:35:10.221 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_auth_use_sts": "false", 2023-10-20T03:35:10.221 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_client_max_sig_ver": "-1", 2023-10-20T03:35:10.222 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_s3_success_create_obj_status": "0", 2023-10-20T03:35:10.222 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_safe_max_objects_per_shard": "102400", 2023-10-20T03:35:10.222 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_scheduler_type": "throttler", 2023-10-20T03:35:10.222 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_script_uri": "", 2023-10-20T03:35:10.222 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_service_provider_name": "", 2023-10-20T03:35:10.222 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_shard_warning_threshold": "90.000000", 2023-10-20T03:35:10.223 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_client_id": "", 2023-10-20T03:35:10.223 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_client_secret": "", 2023-10-20T03:35:10.223 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_entry": "sts", 2023-10-20T03:35:10.223 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_key": "sts", 2023-10-20T03:35:10.223 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_max_session_duration": "43200", 2023-10-20T03:35:10.223 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_min_session_duration": "900", 2023-10-20T03:35:10.224 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sts_token_introspection_url": "", 2023-10-20T03:35:10.224 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_account_in_url": "false", 2023-10-20T03:35:10.224 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_auth_entry": "auth", 2023-10-20T03:35:10.224 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_auth_url": "", 2023-10-20T03:35:10.224 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_custom_header": "", 2023-10-20T03:35:10.225 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_enforce_content_length": "false", 2023-10-20T03:35:10.225 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_need_stats": "true", 2023-10-20T03:35:10.225 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_tenant_name": "", 2023-10-20T03:35:10.225 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_token_expiration": "86400", 2023-10-20T03:35:10.225 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_url": "", 2023-10-20T03:35:10.225 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_url_prefix": "swift", 2023-10-20T03:35:10.226 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_swift_versioning_enabled": "false", 2023-10-20T03:35:10.226 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2023-10-20T03:35:10.226 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2023-10-20T03:35:10.226 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_lease_period": "120", 2023-10-20T03:35:10.226 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2023-10-20T03:35:10.226 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_log_trim_interval": "1200", 2023-10-20T03:35:10.227 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_log_trim_max_buckets": "16", 2023-10-20T03:35:10.227 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2023-10-20T03:35:10.227 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2023-10-20T03:35:10.227 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_obj_etag_verify": "false", 2023-10-20T03:35:10.227 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_trace_history_size": "4096", 2023-10-20T03:35:10.227 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_trace_per_node_log_size": "32", 2023-10-20T03:35:10.228 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2023-10-20T03:35:10.228 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_thread_pool_size": "512", 2023-10-20T03:35:10.228 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_topic_persistency_max_retries": "0", 2023-10-20T03:35:10.228 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_topic_persistency_sleep_duration": "0", 2023-10-20T03:35:10.228 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_topic_persistency_time_to_live": "0", 2023-10-20T03:35:10.228 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_comment": "", 2023-10-20T03:35:10.229 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_createby": "", 2023-10-20T03:35:10.229 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_encoding": "", 2023-10-20T03:35:10.229 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_flag": "false", 2023-10-20T03:35:10.229 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_max_size": "5368709120", 2023-10-20T03:35:10.229 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_origin": "", 2023-10-20T03:35:10.229 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_sha_unit": "524288", 2023-10-20T03:35:10.230 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_torrent_tracker": "", 2023-10-20T03:35:10.230 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_trust_forwarded_https": "false", 2023-10-20T03:35:10.230 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_usage_log_flush_threshold": "1024", 2023-10-20T03:35:10.230 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_usage_log_tick_interval": "30", 2023-10-20T03:35:10.230 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_usage_max_shards": "32", 2023-10-20T03:35:10.231 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_usage_max_user_shards": "1", 2023-10-20T03:35:10.231 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_use_opa_authz": "false", 2023-10-20T03:35:10.231 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_default_quota_max_objects": "-1", 2023-10-20T03:35:10.231 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_default_quota_max_size": "-1", 2023-10-20T03:35:10.231 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_max_buckets": "1000", 2023-10-20T03:35:10.231 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_policies_max_num": "100", 2023-10-20T03:35:10.232 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2023-10-20T03:35:10.232 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_quota_sync_idle_users": "false", 2023-10-20T03:35:10.232 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_quota_sync_interval": "86400", 2023-10-20T03:35:10.232 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_quota_sync_wait_time": "86400", 2023-10-20T03:35:10.232 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_user_unique_email": "true", 2023-10-20T03:35:10.232 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_verify_ssl": "true", 2023-10-20T03:35:10.233 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_website_routing_rules_max_num": "50", 2023-10-20T03:35:10.233 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zone": "", 2023-10-20T03:35:10.233 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zone_id": "", 2023-10-20T03:35:10.233 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zone_root_pool": ".rgw.root", 2023-10-20T03:35:10.233 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zonegroup": "", 2023-10-20T03:35:10.234 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zonegroup_id": "", 2023-10-20T03:35:10.234 INFO:teuthology.orchestra.run.smithi033.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2023-10-20T03:35:10.234 INFO:teuthology.orchestra.run.smithi033.stdout: "rgwlc_auto_session_clear": "true", 2023-10-20T03:35:10.234 INFO:teuthology.orchestra.run.smithi033.stdout: "rgwlc_skip_bucket_step": "false", 2023-10-20T03:35:10.234 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_block_size": "4096", 2023-10-20T03:35:10.234 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_bloom_bits_per_key": "20", 2023-10-20T03:35:10.235 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2023-10-20T03:35:10.235 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2023-10-20T03:35:10.235 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_row_ratio": "0.000000", 2023-10-20T03:35:10.235 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_shard_bits": "4", 2023-10-20T03:35:10.235 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_size": "536870912", 2023-10-20T03:35:10.235 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cache_type": "binned_lru", 2023-10-20T03:35:10.236 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cf_compact_on_deletion": "true", 2023-10-20T03:35:10.236 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2023-10-20T03:35:10.236 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2023-10-20T03:35:10.236 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_collect_compaction_stats": "false", 2023-10-20T03:35:10.236 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_collect_extended_stats": "false", 2023-10-20T03:35:10.236 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_collect_memory_stats": "false", 2023-10-20T03:35:10.237 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_delete_range_threshold": "1048576", 2023-10-20T03:35:10.237 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_index_type": "binary_search", 2023-10-20T03:35:10.237 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_log_to_ceph_log": "true", 2023-10-20T03:35:10.237 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_metadata_block_size": "4096", 2023-10-20T03:35:10.237 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_partition_filters": "false", 2023-10-20T03:35:10.237 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_perf": "false", 2023-10-20T03:35:10.238 INFO:teuthology.orchestra.run.smithi033.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2023-10-20T03:35:10.238 INFO:teuthology.orchestra.run.smithi033.stdout: "rotating_keys_bootstrap_timeout": "30", 2023-10-20T03:35:10.238 INFO:teuthology.orchestra.run.smithi033.stdout: "rotating_keys_renewal_timeout": "10", 2023-10-20T03:35:10.238 INFO:teuthology.orchestra.run.smithi033.stdout: "run_dir": "/var/run/ceph", 2023-10-20T03:35:10.238 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_block_create": "true", 2023-10-20T03:35:10.238 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_cache_lru_size": "67108864", 2023-10-20T03:35:10.239 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_cbjournal_size": "5368709120", 2023-10-20T03:35:10.239 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_default_max_object_size": "16777216", 2023-10-20T03:35:10.239 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_default_object_metadata_reservation": "16777216", 2023-10-20T03:35:10.239 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_device_size": "53687091200", 2023-10-20T03:35:10.239 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_journal_batch_capacity": "16", 2023-10-20T03:35:10.239 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_journal_batch_flush_size": "16777216", 2023-10-20T03:35:10.240 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2023-10-20T03:35:10.240 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_journal_iodepth_limit": "5", 2023-10-20T03:35:10.240 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_main_device_type": "SSD", 2023-10-20T03:35:10.240 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_max_concurrent_transactions": "8", 2023-10-20T03:35:10.240 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2023-10-20T03:35:10.241 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2023-10-20T03:35:10.241 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2023-10-20T03:35:10.241 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_obj_data_write_amplification": "1.250000", 2023-10-20T03:35:10.241 INFO:teuthology.orchestra.run.smithi033.stdout: "seastore_segment_size": "67108864", 2023-10-20T03:35:10.241 INFO:teuthology.orchestra.run.smithi033.stdout: "set_keepcaps": "false", 2023-10-20T03:35:10.241 INFO:teuthology.orchestra.run.smithi033.stdout: "setgroup": "", 2023-10-20T03:35:10.241 INFO:teuthology.orchestra.run.smithi033.stdout: "setuser": "", 2023-10-20T03:35:10.242 INFO:teuthology.orchestra.run.smithi033.stdout: "setuser_match_path": "", 2023-10-20T03:35:10.242 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_misplaced_ratio": "0.050000", 2023-10-20T03:35:10.242 INFO:teuthology.orchestra.run.smithi033.stdout: "thp": "false", 2023-10-20T03:35:10.242 INFO:teuthology.orchestra.run.smithi033.stdout: "threadpool_default_timeout": "60", 2023-10-20T03:35:10.242 INFO:teuthology.orchestra.run.smithi033.stdout: "threadpool_empty_queue_max_wait": "2", 2023-10-20T03:35:10.243 INFO:teuthology.orchestra.run.smithi033.stdout: "throttler_perf_counter": "true" 2023-10-20T03:35:10.243 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:10.245 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2023-10-20T03:35:10.394 INFO:teuthology.orchestra.run.smithi033.stdout:mon.a: {} 2023-10-20T03:35:10.394 INFO:teuthology.orchestra.run.smithi033.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-10-20T03:35:10.407 INFO:teuthology.orchestra.run.smithi033.stdout:mon.b: {} 2023-10-20T03:35:10.407 INFO:teuthology.orchestra.run.smithi033.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-10-20T03:35:10.420 INFO:teuthology.orchestra.run.smithi033.stdout:mon.c: {} 2023-10-20T03:35:10.420 INFO:teuthology.orchestra.run.smithi033.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-10-20T03:35:10.436 DEBUG:teuthology.orchestra.run.smithi040:> type ceph-objectstore-tool 2023-10-20T03:35:10.442 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-10-20T03:35:10.442 DEBUG:teuthology.orchestra.run.smithi170:> type ceph-objectstore-tool 2023-10-20T03:35:10.448 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-10-20T03:35:10.448 DEBUG:teuthology.orchestra.run.smithi064:> type ceph-objectstore-tool 2023-10-20T03:35:10.454 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-10-20T03:35:10.454 DEBUG:teuthology.orchestra.run.smithi033:> type ceph-objectstore-tool 2023-10-20T03:35:10.459 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-10-20T03:35:10.459 INFO:teuthology.run_tasks:Running task rados... 2023-10-20T03:35:10.469 INFO:tasks.rados:Beginning rados... 2023-10-20T03:35:10.470 DEBUG:teuthology.run_tasks:Unwinding manager rados 2023-10-20T03:35:10.478 INFO:tasks.rados:joining rados 2023-10-20T03:35:10.478 INFO:tasks.thrashosds.thrasher:starting do_thrash 2023-10-20T03:35:10.478 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-10-20T03:35:10.479 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-10-20T03:35:10.479 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-10-20T03:35:10.479 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-20T03:35:10.479 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:35:10.480 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2023-10-20T03:35:10.480 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2023-10-20T03:35:10.480 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:35:10.481 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2023-10-20T03:35:10.481 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-10-20T03:35:10.482 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2023-10-20T03:35:10.482 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-10-20T03:35:10.482 INFO:tasks.rados:clients are ['client.0'] 2023-10-20T03:35:10.483 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2023-10-20T03:35:10.502 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:10.502+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:10.559 INFO:tasks.ceph.osd.6:Stopped 2023-10-20T03:35:10.560 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2023-10-20T03:35:10.560 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi064.front.sepia.ceph.com 2023-10-20T03:35:10.560 DEBUG:teuthology.orchestra.run.smithi064:> type ceph-objectstore-tool 2023-10-20T03:35:10.565 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-10-20T03:35:10.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2023-10-20T03:35:10.608 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:10.605+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:10.629 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:10.629 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:10.629 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:10.629 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:10.651 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-10-20T03:35:10.654 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:10.654 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:35:10.683 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:10.677+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:10.689 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:10.689 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:35:10.722 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:10.723 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:35:10.757 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:10.757 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:35:10.765 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:10.765 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:10.778 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-10-20T03:35:10.787 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:10.788 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:35:10.788 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:10.786+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:10.822 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:10.822 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:35:10.885 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:10.881+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:10.901 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:10.901 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:10.915 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-10-20T03:35:10.986 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:10.981+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:11.080 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:11.080 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:11.086 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:11.083+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:11.094 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-10-20T03:35:11.158 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2023-10-20T03:35:11.158 INFO:teuthology.orchestra.run.smithi033.stderr:load: shec 2023-10-20T03:35:11.186 INFO:tasks.rados:starting run 0 out of 1 2023-10-20T03:35:11.187 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2023-10-20T03:35:11.187 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2023-10-20T03:35:11.187 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:11.186+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:11.227 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:11.228 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:11.243 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-10-20T03:35:11.289 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:11.290+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:11.389 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:11.385+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:11.411 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:11.411 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:11.424 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-10-20T03:35:11.489 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:11.486+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:11.547 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:11.547 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:11.548 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:11.548 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:11.561 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-10-20T03:35:11.606 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:11.606+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:11.691 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:11.691+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:11.730 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:11.730 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:11.731 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:35:11.731 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:35:11.731 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:11.743 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-10-20T03:35:11.791 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:11.789+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:11.807 INFO:teuthology.orchestra.run.smithi064.stdout:1.0 2023-10-20T03:35:11.867 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:11.867 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:11.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:11.868 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [] 2023-10-20T03:35:11.868 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:11.881 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-10-20T03:35:11.892 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:11.891+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:11.993 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:11.989+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:12.008 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:12.009 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:12.009 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:12.009 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:12.024 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-10-20T03:35:12.094 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:12.089+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:12.152 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:12.153 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:12.153 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:35:12.153 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:35:12.153 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:12.168 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-10-20T03:35:12.195 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:12.193+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:12.293 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:12.293 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:12.294 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:12.294 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [] 2023-10-20T03:35:12.294 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:12.296 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:12.293+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:12.309 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-10-20T03:35:12.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op export --pgid 1.0 --file /var/log/ceph/exp.1.0.6 2023-10-20T03:35:12.396 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:12.394+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:12.443 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:12.443 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:12.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-10-20T03:35:12.497 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:12.498+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:12.578 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:12.578 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:12.594 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-10-20T03:35:12.598 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:12.593+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:12.699 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:12.699+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:12.719 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:12.719 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:12.734 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-10-20T03:35:12.799 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:12.797+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:12.860 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:12.860 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:12.875 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-10-20T03:35:12.919 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:12.917+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:13.001 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:12.997+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:13.038 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:13.038 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:13.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-10-20T03:35:13.102 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:13.099+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:13.182 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-10-20T03:35:13.183 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_0' created 2023-10-20T03:35:13.203 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:13.203+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:13.220 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2023-10-20T03:35:13.233 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:13.233 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:13.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-10-20T03:35:13.306 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:13.306+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:13.374 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:13.374 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:13.374 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:35:13.375 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:13.389 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-10-20T03:35:13.434 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:13.433+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:13.506 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:13.507+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:13.513 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:13.514 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:13.514 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:35:13.514 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:35:13.514 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:13.528 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-10-20T03:35:13.597 INFO:teuthology.orchestra.run.smithi064.stderr:Exporting 1.0 info 1.0( v 26'157 (0'0,26'157] local-lis/les=23/24 n=2 ec=23/23 lis/c=23/23 les/c/f=24/24/0 sis=23) 2023-10-20T03:35:13.607 INFO:teuthology.orchestra.run.smithi064.stderr:Read #1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head# 2023-10-20T03:35:13.607 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:13.606+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:13.608 INFO:teuthology.orchestra.run.smithi064.stderr:Read #1:30306672:devicehealth::main.db.0000000000000000:head# 2023-10-20T03:35:13.610 INFO:teuthology.orchestra.run.smithi064.stderr:Export successful 2023-10-20T03:35:13.698 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:13.698 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:35:13.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:35:13.698 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2023-10-20T03:35:13.699 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:13.708 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:13.707+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:13.711 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-10-20T03:35:13.809 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:13.807+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:13.837 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:13.838 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:13.838 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:35:13.838 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:13.852 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-10-20T03:35:13.910 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:13.910+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:13.977 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:13.977 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:13.978 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:35:13.978 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:35:13.978 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:13.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-10-20T03:35:14.039 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:14.037+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:14.111 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:14.111 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:35:14.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:35:14.111 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2023-10-20T03:35:14.112 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:14.112 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:14.110+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:14.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-10-20T03:35:14.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --force --op remove --pgid 1.0 2023-10-20T03:35:14.193 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2023-10-20T03:35:14.193 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rados' on pool 'unique_pool_0' 2023-10-20T03:35:14.213 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:14.210+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:14.269 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:14.269 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:14.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-10-20T03:35:14.313 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:14.311+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:14.405 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:14.406 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:14.414 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:14.409+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:14.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-10-20T03:35:14.514 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:14.509+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:14.546 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:14.546 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:14.562 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-10-20T03:35:14.616 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:14.615+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:14.685 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:14.686 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:14.686 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2023-10-20T03:35:14.686 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:14.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-10-20T03:35:14.716 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:14.715+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:14.817 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:14.813+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:14.824 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:14.824 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2023-10-20T03:35:14.825 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2023-10-20T03:35:14.825 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:14.839 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-10-20T03:35:14.918 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:14.918+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:14.964 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2023-10-20T03:35:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2023-10-20T03:35:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2023-10-20T03:35:14.965 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:14.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-10-20T03:35:15.019 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:15.017+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:15.108 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:15.109 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:15.109 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2023-10-20T03:35:15.109 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:15.119 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:15.117+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:15.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-10-20T03:35:15.221 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:15.218+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:15.236 DEBUG:teuthology.orchestra.run.smithi170:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2023-10-20T03:35:15.260 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight read -> 100 2023-10-20T03:35:15.260 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight delete -> 50 2023-10-20T03:35:15.260 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight snap_create -> 50 2023-10-20T03:35:15.260 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight snap_remove -> 50 2023-10-20T03:35:15.261 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight rollback -> 50 2023-10-20T03:35:15.261 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight setattr -> 25 2023-10-20T03:35:15.261 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight rmattr -> 25 2023-10-20T03:35:15.261 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight copy_from -> 50 2023-10-20T03:35:15.261 INFO:tasks.rados.rados.0.smithi170.stdout:adding op weight append -> 100 2023-10-20T03:35:15.262 INFO:tasks.rados.rados.0.smithi170.stdout:ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 2023-10-20T03:35:15.262 INFO:tasks.rados.rados.0.smithi170.stdout:Configuration: 2023-10-20T03:35:15.262 INFO:tasks.rados.rados.0.smithi170.stdout: Number of operations: 400 2023-10-20T03:35:15.262 INFO:tasks.rados.rados.0.smithi170.stdout: Number of objects: 50 2023-10-20T03:35:15.262 INFO:tasks.rados.rados.0.smithi170.stdout: Max in flight operations: 16 2023-10-20T03:35:15.263 INFO:tasks.rados.rados.0.smithi170.stdout: Object size (in bytes): 4000000 2023-10-20T03:35:15.263 INFO:tasks.rados.rados.0.smithi170.stdout: Write stride min: 400000 2023-10-20T03:35:15.263 INFO:tasks.rados.rados.0.smithi170.stdout: Write stride max: 800000 2023-10-20T03:35:15.288 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:15.288 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:15.288 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2023-10-20T03:35:15.288 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2023-10-20T03:35:15.289 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:15.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-10-20T03:35:15.321 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:15.322+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:15.422 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:15.422+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:15.431 INFO:teuthology.orchestra.run.smithi064.stdout: marking collection for removal 2023-10-20T03:35:15.431 INFO:teuthology.orchestra.run.smithi064.stdout:setting '_remove' omap key 2023-10-20T03:35:15.431 INFO:teuthology.orchestra.run.smithi064.stdout:finish_remove_pgs 1.0_head removing 1.0 2023-10-20T03:35:15.433 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:15.434 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2023-10-20T03:35:15.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2023-10-20T03:35:15.434 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2023-10-20T03:35:15.435 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:15.441 INFO:teuthology.orchestra.run.smithi064.stdout:Remove successful 2023-10-20T03:35:15.450 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-10-20T03:35:15.523 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:15.517+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:15.579 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:15.580 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:15.593 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-10-20T03:35:15.624 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:15.622+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:15.627 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:15.646 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:15.646 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:35:15.673 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:15.674 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:35:15.706 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:15.706 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-10-20T03:35:15.713 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:15.713 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:15.725 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:15.723+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:15.727 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-10-20T03:35:15.738 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:15.739 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:35:15.767 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:15.768 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:35:15.795 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:15.796 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:35:15.824 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:15.824 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:35:15.825 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:15.825+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:15.852 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:15.853 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-10-20T03:35:15.880 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:15.880 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:35:15.886 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:15.887 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:15.893 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:15.893 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:15.907 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-10-20T03:35:15.955 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:15.955+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:15.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.0.6 2023-10-20T03:35:16.029 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:16.029 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:16.043 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-10-20T03:35:16.051 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:16.049+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:16.128 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:16.126+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:16.208 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:16.208 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:16.213 INFO:tasks.rados.rados.0.smithi170.stdout:1: write initial oid 1 2023-10-20T03:35:16.213 INFO:tasks.rados.rados.0.smithi170.stdout:1: seq_num 0 ranges {0=32768} 2023-10-20T03:35:16.214 INFO:tasks.rados.rados.0.smithi170.stdout:1: writing smithi17039300-1 from 0 to 32768 tid 1 2023-10-20T03:35:16.214 INFO:tasks.rados.rados.0.smithi170.stdout:2: write initial oid 2 2023-10-20T03:35:16.215 INFO:tasks.rados.rados.0.smithi170.stdout:2: seq_num 1 ranges {0=32768} 2023-10-20T03:35:16.215 INFO:tasks.rados.rados.0.smithi170.stdout:2: writing smithi17039300-2 from 0 to 32768 tid 1 2023-10-20T03:35:16.215 INFO:tasks.rados.rados.0.smithi170.stdout:3: write initial oid 3 2023-10-20T03:35:16.215 INFO:tasks.rados.rados.0.smithi170.stdout:3: seq_num 2 ranges {0=49152} 2023-10-20T03:35:16.215 INFO:tasks.rados.rados.0.smithi170.stdout:3: writing smithi17039300-3 from 0 to 49152 tid 1 2023-10-20T03:35:16.216 INFO:tasks.rados.rados.0.smithi170.stdout:4: write initial oid 4 2023-10-20T03:35:16.216 INFO:tasks.rados.rados.0.smithi170.stdout:4: seq_num 3 ranges {0=65536} 2023-10-20T03:35:16.216 INFO:tasks.rados.rados.0.smithi170.stdout:4: writing smithi17039300-4 from 0 to 65536 tid 1 2023-10-20T03:35:16.217 INFO:tasks.rados.rados.0.smithi170.stdout:5: write initial oid 5 2023-10-20T03:35:16.217 INFO:tasks.rados.rados.0.smithi170.stdout:5: seq_num 4 ranges {0=81920} 2023-10-20T03:35:16.220 INFO:tasks.rados.rados.0.smithi170.stdout:5: writing smithi17039300-5 from 0 to 81920 tid 1 2023-10-20T03:35:16.220 INFO:tasks.rados.rados.0.smithi170.stdout:6: write initial oid 6 2023-10-20T03:35:16.220 INFO:tasks.rados.rados.0.smithi170.stdout:6: seq_num 5 ranges {0=98304} 2023-10-20T03:35:16.221 INFO:tasks.rados.rados.0.smithi170.stdout:6: writing smithi17039300-6 from 0 to 98304 tid 1 2023-10-20T03:35:16.221 INFO:tasks.rados.rados.0.smithi170.stdout:7: write initial oid 7 2023-10-20T03:35:16.221 INFO:tasks.rados.rados.0.smithi170.stdout:7: seq_num 6 ranges {0=114688} 2023-10-20T03:35:16.221 INFO:tasks.rados.rados.0.smithi170.stdout:7: writing smithi17039300-7 from 0 to 114688 tid 1 2023-10-20T03:35:16.221 INFO:tasks.rados.rados.0.smithi170.stdout:8: write initial oid 8 2023-10-20T03:35:16.221 INFO:tasks.rados.rados.0.smithi170.stdout:8: seq_num 7 ranges {0=131072} 2023-10-20T03:35:16.222 INFO:tasks.rados.rados.0.smithi170.stdout:8: writing smithi17039300-8 from 0 to 131072 tid 1 2023-10-20T03:35:16.222 INFO:tasks.rados.rados.0.smithi170.stdout:9: write initial oid 9 2023-10-20T03:35:16.222 INFO:tasks.rados.rados.0.smithi170.stdout:9: seq_num 8 ranges {0=147456} 2023-10-20T03:35:16.223 INFO:tasks.rados.rados.0.smithi170.stdout:9: writing smithi17039300-9 from 0 to 147456 tid 1 2023-10-20T03:35:16.223 INFO:tasks.rados.rados.0.smithi170.stdout:10: write initial oid 10 2023-10-20T03:35:16.223 INFO:tasks.rados.rados.0.smithi170.stdout:10: seq_num 9 ranges {0=163840} 2023-10-20T03:35:16.224 INFO:tasks.rados.rados.0.smithi170.stdout:10: writing smithi17039300-10 from 0 to 163840 tid 1 2023-10-20T03:35:16.224 INFO:tasks.rados.rados.0.smithi170.stdout:11: write initial oid 11 2023-10-20T03:35:16.225 INFO:tasks.rados.rados.0.smithi170.stdout:11: seq_num 10 ranges {0=180224} 2023-10-20T03:35:16.226 INFO:tasks.rados.rados.0.smithi170.stdout:11: writing smithi17039300-11 from 0 to 180224 tid 1 2023-10-20T03:35:16.226 INFO:tasks.rados.rados.0.smithi170.stdout:12: write initial oid 12 2023-10-20T03:35:16.227 INFO:tasks.rados.rados.0.smithi170.stdout:12: seq_num 11 ranges {0=196608} 2023-10-20T03:35:16.227 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-10-20T03:35:16.228 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-10-20T03:35:16.230 INFO:tasks.rados.rados.0.smithi170.stdout:12: writing smithi17039300-12 from 0 to 196608 tid 1 2023-10-20T03:35:16.230 INFO:tasks.rados.rados.0.smithi170.stdout:2: finishing write tid 1 to smithi17039300-2 2023-10-20T03:35:16.231 INFO:tasks.rados.rados.0.smithi170.stdout:3: finishing write tid 1 to smithi17039300-3 2023-10-20T03:35:16.231 INFO:tasks.rados.rados.0.smithi170.stdout:13: write initial oid 13 2023-10-20T03:35:16.231 INFO:tasks.rados.rados.0.smithi170.stdout:13: seq_num 12 ranges {0=212992} 2023-10-20T03:35:16.231 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:16.227+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:16.233 INFO:tasks.rados.rados.0.smithi170.stdout:13: writing smithi17039300-13 from 0 to 212992 tid 1 2023-10-20T03:35:16.233 INFO:tasks.rados.rados.0.smithi170.stdout:14: write initial oid 14 2023-10-20T03:35:16.233 INFO:tasks.rados.rados.0.smithi170.stdout:14: seq_num 13 ranges {0=229376} 2023-10-20T03:35:16.233 INFO:tasks.rados.rados.0.smithi170.stdout:14: writing smithi17039300-14 from 0 to 229376 tid 1 2023-10-20T03:35:16.234 INFO:tasks.rados.rados.0.smithi170.stdout:15: write initial oid 15 2023-10-20T03:35:16.234 INFO:tasks.rados.rados.0.smithi170.stdout:15: seq_num 14 ranges {0=245760} 2023-10-20T03:35:16.235 INFO:tasks.rados.rados.0.smithi170.stdout:15: writing smithi17039300-15 from 0 to 245760 tid 1 2023-10-20T03:35:16.235 INFO:tasks.rados.rados.0.smithi170.stdout:16: write initial oid 16 2023-10-20T03:35:16.235 INFO:tasks.rados.rados.0.smithi170.stdout:16: seq_num 15 ranges {0=262144} 2023-10-20T03:35:16.236 INFO:tasks.rados.rados.0.smithi170.stdout:16: writing smithi17039300-16 from 0 to 262144 tid 1 2023-10-20T03:35:16.236 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2023-10-20T03:35:16.237 INFO:tasks.rados.rados.0.smithi170.stdout:2: finishing write tid 2 to smithi17039300-2 2023-10-20T03:35:16.237 INFO:tasks.rados.rados.0.smithi170.stdout:1: finishing write tid 1 to smithi17039300-1 2023-10-20T03:35:16.238 INFO:tasks.rados.rados.0.smithi170.stdout:2: finishing write tid 3 to smithi17039300-2 2023-10-20T03:35:16.238 INFO:tasks.rados.rados.0.smithi170.stdout:2: oid 2 updating version 0 to 1 2023-10-20T03:35:16.238 INFO:tasks.rados.rados.0.smithi170.stdout:2: oid 2 updating version 1 to 2 2023-10-20T03:35:16.239 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2023-10-20T03:35:16.239 INFO:tasks.rados.rados.0.smithi170.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2023-10-20T03:35:16.239 INFO:tasks.rados.rados.0.smithi170.stdout:2: done (15 left) 2023-10-20T03:35:16.239 INFO:tasks.rados.rados.0.smithi170.stdout:17: write initial oid 17 2023-10-20T03:35:16.239 INFO:tasks.rados.rados.0.smithi170.stdout:17: seq_num 16 ranges {0=278528} 2023-10-20T03:35:16.240 INFO:tasks.rados.rados.0.smithi170.stdout:17: writing smithi17039300-17 from 0 to 278528 tid 1 2023-10-20T03:35:16.240 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2023-10-20T03:35:16.241 INFO:tasks.rados.rados.0.smithi170.stdout:1: finishing write tid 2 to smithi17039300-1 2023-10-20T03:35:16.259 INFO:tasks.rados.rados.0.smithi170.stdout:1: finishing write tid 3 to smithi17039300-1 2023-10-20T03:35:16.260 INFO:tasks.rados.rados.0.smithi170.stdout:1: oid 1 updating version 0 to 1 2023-10-20T03:35:16.260 INFO:tasks.rados.rados.0.smithi170.stdout:1: oid 1 updating version 1 to 2 2023-10-20T03:35:16.260 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2023-10-20T03:35:16.260 INFO:tasks.rados.rados.0.smithi170.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2023-10-20T03:35:16.260 INFO:tasks.rados.rados.0.smithi170.stdout:1: done (15 left) 2023-10-20T03:35:16.261 INFO:tasks.rados.rados.0.smithi170.stdout:18: write initial oid 18 2023-10-20T03:35:16.261 INFO:tasks.rados.rados.0.smithi170.stdout:18: seq_num 17 ranges {0=294912} 2023-10-20T03:35:16.261 INFO:tasks.rados.rados.0.smithi170.stdout:18: writing smithi17039300-18 from 0 to 294912 tid 1 2023-10-20T03:35:16.262 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2023-10-20T03:35:16.262 INFO:tasks.rados.rados.0.smithi170.stdout:5: finishing write tid 1 to smithi17039300-5 2023-10-20T03:35:16.262 INFO:tasks.rados.rados.0.smithi170.stdout:5: finishing write tid 2 to smithi17039300-5 2023-10-20T03:35:16.262 INFO:tasks.rados.rados.0.smithi170.stdout:7: finishing write tid 1 to smithi17039300-7 2023-10-20T03:35:16.263 INFO:tasks.rados.rados.0.smithi170.stdout:7: finishing write tid 2 to smithi17039300-7 2023-10-20T03:35:16.263 INFO:tasks.rados.rados.0.smithi170.stdout:8: finishing write tid 1 to smithi17039300-8 2023-10-20T03:35:16.266 INFO:tasks.rados.rados.0.smithi170.stdout:6: finishing write tid 1 to smithi17039300-6 2023-10-20T03:35:16.267 INFO:tasks.rados.rados.0.smithi170.stdout:13: finishing write tid 1 to smithi17039300-13 2023-10-20T03:35:16.268 INFO:tasks.rados.rados.0.smithi170.stdout:3: finishing write tid 2 to smithi17039300-3 2023-10-20T03:35:16.268 INFO:tasks.rados.rados.0.smithi170.stdout:12: finishing write tid 1 to smithi17039300-12 2023-10-20T03:35:16.268 INFO:tasks.rados.rados.0.smithi170.stdout:12: finishing write tid 2 to smithi17039300-12 2023-10-20T03:35:16.269 INFO:tasks.rados.rados.0.smithi170.stdout:4: finishing write tid 1 to smithi17039300-4 2023-10-20T03:35:16.269 INFO:tasks.rados.rados.0.smithi170.stdout:4: finishing write tid 2 to smithi17039300-4 2023-10-20T03:35:16.269 INFO:tasks.rados.rados.0.smithi170.stdout:5: finishing write tid 3 to smithi17039300-5 2023-10-20T03:35:16.269 INFO:tasks.rados.rados.0.smithi170.stdout:5: oid 5 updating version 0 to 1 2023-10-20T03:35:16.270 INFO:tasks.rados.rados.0.smithi170.stdout:5: oid 5 updating version 1 to 2 2023-10-20T03:35:16.270 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2023-10-20T03:35:16.270 INFO:tasks.rados.rados.0.smithi170.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2023-10-20T03:35:16.270 INFO:tasks.rados.rados.0.smithi170.stdout:5: done (15 left) 2023-10-20T03:35:16.270 INFO:tasks.rados.rados.0.smithi170.stdout:19: write initial oid 19 2023-10-20T03:35:16.271 INFO:tasks.rados.rados.0.smithi170.stdout:19: seq_num 18 ranges {0=311296} 2023-10-20T03:35:16.272 INFO:tasks.rados.rados.0.smithi170.stdout:19: writing smithi17039300-19 from 0 to 311296 tid 1 2023-10-20T03:35:16.272 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2023-10-20T03:35:16.272 INFO:tasks.rados.rados.0.smithi170.stdout:9: finishing write tid 1 to smithi17039300-9 2023-10-20T03:35:16.272 INFO:tasks.rados.rados.0.smithi170.stdout:15: finishing write tid 1 to smithi17039300-15 2023-10-20T03:35:16.273 INFO:tasks.rados.rados.0.smithi170.stdout:15: finishing write tid 2 to smithi17039300-15 2023-10-20T03:35:16.273 INFO:tasks.rados.rados.0.smithi170.stdout:16: finishing write tid 1 to smithi17039300-16 2023-10-20T03:35:16.273 INFO:tasks.rados.rados.0.smithi170.stdout:9: finishing write tid 2 to smithi17039300-9 2023-10-20T03:35:16.273 INFO:tasks.rados.rados.0.smithi170.stdout:7: finishing write tid 3 to smithi17039300-7 2023-10-20T03:35:16.273 INFO:tasks.rados.rados.0.smithi170.stdout:7: oid 7 updating version 0 to 3 2023-10-20T03:35:16.274 INFO:tasks.rados.rados.0.smithi170.stdout:7: oid 7 updating version 3 to 4 2023-10-20T03:35:16.274 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 7 v 4 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-10-20T03:35:16.274 INFO:tasks.rados.rados.0.smithi170.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2023-10-20T03:35:16.274 INFO:tasks.rados.rados.0.smithi170.stdout:7: done (15 left) 2023-10-20T03:35:16.274 INFO:tasks.rados.rados.0.smithi170.stdout:20: write initial oid 20 2023-10-20T03:35:16.274 INFO:tasks.rados.rados.0.smithi170.stdout:20: seq_num 19 ranges {0=327680} 2023-10-20T03:35:16.276 INFO:tasks.rados.rados.0.smithi170.stdout:20: writing smithi17039300-20 from 0 to 327680 tid 1 2023-10-20T03:35:16.276 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2023-10-20T03:35:16.276 INFO:tasks.rados.rados.0.smithi170.stdout:6: finishing write tid 2 to smithi17039300-6 2023-10-20T03:35:16.276 INFO:tasks.rados.rados.0.smithi170.stdout:4: finishing write tid 3 to smithi17039300-4 2023-10-20T03:35:16.276 INFO:tasks.rados.rados.0.smithi170.stdout:4: oid 4 updating version 0 to 2 2023-10-20T03:35:16.277 INFO:tasks.rados.rados.0.smithi170.stdout:4: oid 4 version 2 is already newer than 1 2023-10-20T03:35:16.277 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2023-10-20T03:35:16.277 INFO:tasks.rados.rados.0.smithi170.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2023-10-20T03:35:16.277 INFO:tasks.rados.rados.0.smithi170.stdout:16: finishing write tid 2 to smithi17039300-16 2023-10-20T03:35:16.277 INFO:tasks.rados.rados.0.smithi170.stdout:12: finishing write tid 3 to smithi17039300-12 2023-10-20T03:35:16.277 INFO:tasks.rados.rados.0.smithi170.stdout:12: oid 12 updating version 0 to 2 2023-10-20T03:35:16.278 INFO:tasks.rados.rados.0.smithi170.stdout:12: oid 12 version 2 is already newer than 1 2023-10-20T03:35:16.278 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 12 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2023-10-20T03:35:16.278 INFO:tasks.rados.rados.0.smithi170.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2023-10-20T03:35:16.278 INFO:tasks.rados.rados.0.smithi170.stdout:4: done (15 left) 2023-10-20T03:35:16.278 INFO:tasks.rados.rados.0.smithi170.stdout:12: done (14 left) 2023-10-20T03:35:16.279 INFO:tasks.rados.rados.0.smithi170.stdout:21: write initial oid 21 2023-10-20T03:35:16.279 INFO:tasks.rados.rados.0.smithi170.stdout:21: seq_num 20 ranges {0=344064} 2023-10-20T03:35:16.280 INFO:tasks.rados.rados.0.smithi170.stdout:21: writing smithi17039300-21 from 0 to 344064 tid 1 2023-10-20T03:35:16.280 INFO:tasks.rados.rados.0.smithi170.stdout:22: write initial oid 22 2023-10-20T03:35:16.281 INFO:tasks.rados.rados.0.smithi170.stdout:22: seq_num 21 ranges {0=360448} 2023-10-20T03:35:16.281 INFO:tasks.rados.rados.0.smithi170.stdout:22: writing smithi17039300-22 from 0 to 360448 tid 1 2023-10-20T03:35:16.281 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2023-10-20T03:35:16.281 INFO:tasks.rados.rados.0.smithi170.stdout:10: finishing write tid 1 to smithi17039300-10 2023-10-20T03:35:16.281 INFO:tasks.rados.rados.0.smithi170.stdout:13: finishing write tid 2 to smithi17039300-13 2023-10-20T03:35:16.281 INFO:tasks.rados.rados.0.smithi170.stdout:10: finishing write tid 2 to smithi17039300-10 2023-10-20T03:35:16.282 INFO:tasks.rados.rados.0.smithi170.stdout:3: finishing write tid 3 to smithi17039300-3 2023-10-20T03:35:16.282 INFO:tasks.rados.rados.0.smithi170.stdout:3: oid 3 updating version 0 to 2 2023-10-20T03:35:16.282 INFO:tasks.rados.rados.0.smithi170.stdout:3: oid 3 version 2 is already newer than 1 2023-10-20T03:35:16.282 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2023-10-20T03:35:16.282 INFO:tasks.rados.rados.0.smithi170.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2023-10-20T03:35:16.283 INFO:tasks.rados.rados.0.smithi170.stdout:6: finishing write tid 3 to smithi17039300-6 2023-10-20T03:35:16.283 INFO:tasks.rados.rados.0.smithi170.stdout:6: oid 6 updating version 0 to 1 2023-10-20T03:35:16.283 INFO:tasks.rados.rados.0.smithi170.stdout:6: oid 6 updating version 1 to 2 2023-10-20T03:35:16.283 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2023-10-20T03:35:16.283 INFO:tasks.rados.rados.0.smithi170.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2023-10-20T03:35:16.284 INFO:tasks.rados.rados.0.smithi170.stdout:3: done (15 left) 2023-10-20T03:35:16.284 INFO:tasks.rados.rados.0.smithi170.stdout:6: done (14 left) 2023-10-20T03:35:16.284 INFO:tasks.rados.rados.0.smithi170.stdout:23: write initial oid 23 2023-10-20T03:35:16.284 INFO:tasks.rados.rados.0.smithi170.stdout:23: seq_num 22 ranges {0=376832} 2023-10-20T03:35:16.285 INFO:tasks.rados.rados.0.smithi170.stdout:23: writing smithi17039300-23 from 0 to 376832 tid 1 2023-10-20T03:35:16.286 INFO:tasks.rados.rados.0.smithi170.stdout:13: finishing write tid 3 to smithi17039300-13 2023-10-20T03:35:16.286 INFO:tasks.rados.rados.0.smithi170.stdout:13: oid 13 updating version 0 to 2 2023-10-20T03:35:16.286 INFO:tasks.rados.rados.0.smithi170.stdout:13: oid 13 version 2 is already newer than 1 2023-10-20T03:35:16.286 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 13 v 2 (ObjNum 12 snap 0 seq_num 12) dirty exists 2023-10-20T03:35:16.286 INFO:tasks.rados.rados.0.smithi170.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2023-10-20T03:35:16.287 INFO:tasks.rados.rados.0.smithi170.stdout:15: finishing write tid 3 to smithi17039300-15 2023-10-20T03:35:16.287 INFO:tasks.rados.rados.0.smithi170.stdout:15: oid 15 updating version 0 to 1 2023-10-20T03:35:16.287 INFO:tasks.rados.rados.0.smithi170.stdout:15: oid 15 updating version 1 to 2 2023-10-20T03:35:16.287 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 15 v 2 (ObjNum 14 snap 0 seq_num 14) dirty exists 2023-10-20T03:35:16.287 INFO:tasks.rados.rados.0.smithi170.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2023-10-20T03:35:16.288 INFO:tasks.rados.rados.0.smithi170.stdout:18: finishing write tid 1 to smithi17039300-18 2023-10-20T03:35:16.288 INFO:tasks.rados.rados.0.smithi170.stdout:9: finishing write tid 3 to smithi17039300-9 2023-10-20T03:35:16.288 INFO:tasks.rados.rados.0.smithi170.stdout:9: oid 9 updating version 0 to 1 2023-10-20T03:35:16.288 INFO:tasks.rados.rados.0.smithi170.stdout:9: oid 9 updating version 1 to 2 2023-10-20T03:35:16.288 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2023-10-20T03:35:16.288 INFO:tasks.rados.rados.0.smithi170.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2023-10-20T03:35:16.289 INFO:tasks.rados.rados.0.smithi170.stdout:18: finishing write tid 2 to smithi17039300-18 2023-10-20T03:35:16.289 INFO:tasks.rados.rados.0.smithi170.stdout:9: done (14 left) 2023-10-20T03:35:16.289 INFO:tasks.rados.rados.0.smithi170.stdout:13: done (13 left) 2023-10-20T03:35:16.289 INFO:tasks.rados.rados.0.smithi170.stdout:15: done (12 left) 2023-10-20T03:35:16.289 INFO:tasks.rados.rados.0.smithi170.stdout:24: write initial oid 24 2023-10-20T03:35:16.290 INFO:tasks.rados.rados.0.smithi170.stdout:24: seq_num 23 ranges {0=393216} 2023-10-20T03:35:16.290 INFO:tasks.rados.rados.0.smithi170.stdout:24: writing smithi17039300-24 from 0 to 393216 tid 1 2023-10-20T03:35:16.290 INFO:tasks.rados.rados.0.smithi170.stdout:10: finishing write tid 3 to smithi17039300-10 2023-10-20T03:35:16.290 INFO:tasks.rados.rados.0.smithi170.stdout:10: oid 10 updating version 0 to 3 2023-10-20T03:35:16.290 INFO:tasks.rados.rados.0.smithi170.stdout:10: oid 10 updating version 3 to 4 2023-10-20T03:35:16.291 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 10 v 4 (ObjNum 9 snap 0 seq_num 9) dirty exists 2023-10-20T03:35:16.291 INFO:tasks.rados.rados.0.smithi170.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2023-10-20T03:35:16.291 INFO:tasks.rados.rados.0.smithi170.stdout:10: done (12 left) 2023-10-20T03:35:16.291 INFO:tasks.rados.rados.0.smithi170.stdout:25: write initial oid 25 2023-10-20T03:35:16.291 INFO:tasks.rados.rados.0.smithi170.stdout:25: seq_num 24 ranges {0=409600} 2023-10-20T03:35:16.292 INFO:tasks.rados.rados.0.smithi170.stdout:25: writing smithi17039300-25 from 0 to 409600 tid 1 2023-10-20T03:35:16.292 INFO:tasks.rados.rados.0.smithi170.stdout:26: write initial oid 26 2023-10-20T03:35:16.293 INFO:tasks.rados.rados.0.smithi170.stdout:26: seq_num 25 ranges {0=425984} 2023-10-20T03:35:16.293 INFO:tasks.rados.rados.0.smithi170.stdout:26: writing smithi17039300-26 from 0 to 425984 tid 1 2023-10-20T03:35:16.293 INFO:tasks.rados.rados.0.smithi170.stdout:27: write initial oid 27 2023-10-20T03:35:16.293 INFO:tasks.rados.rados.0.smithi170.stdout:27: seq_num 26 ranges {0=442368} 2023-10-20T03:35:16.295 INFO:tasks.rados.rados.0.smithi170.stdout:27: writing smithi17039300-27 from 0 to 442368 tid 1 2023-10-20T03:35:16.296 INFO:tasks.rados.rados.0.smithi170.stdout:14: finishing write tid 1 to smithi17039300-14 2023-10-20T03:35:16.296 INFO:tasks.rados.rados.0.smithi170.stdout:14: finishing write tid 2 to smithi17039300-14 2023-10-20T03:35:16.296 INFO:tasks.rados.rados.0.smithi170.stdout:11: finishing write tid 1 to smithi17039300-11 2023-10-20T03:35:16.296 INFO:tasks.rados.rados.0.smithi170.stdout:11: finishing write tid 2 to smithi17039300-11 2023-10-20T03:35:16.296 INFO:tasks.rados.rados.0.smithi170.stdout:17: finishing write tid 1 to smithi17039300-17 2023-10-20T03:35:16.297 INFO:tasks.rados.rados.0.smithi170.stdout:8: finishing write tid 2 to smithi17039300-8 2023-10-20T03:35:16.297 INFO:tasks.rados.rados.0.smithi170.stdout:17: finishing write tid 2 to smithi17039300-17 2023-10-20T03:35:16.297 INFO:tasks.rados.rados.0.smithi170.stdout:18: finishing write tid 3 to smithi17039300-18 2023-10-20T03:35:16.297 INFO:tasks.rados.rados.0.smithi170.stdout:18: oid 18 updating version 0 to 2 2023-10-20T03:35:16.297 INFO:tasks.rados.rados.0.smithi170.stdout:18: oid 18 version 2 is already newer than 1 2023-10-20T03:35:16.298 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 18 v 2 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-10-20T03:35:16.298 INFO:tasks.rados.rados.0.smithi170.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2023-10-20T03:35:16.298 INFO:tasks.rados.rados.0.smithi170.stdout:18: done (14 left) 2023-10-20T03:35:16.298 INFO:tasks.rados.rados.0.smithi170.stdout:28: write initial oid 28 2023-10-20T03:35:16.298 INFO:tasks.rados.rados.0.smithi170.stdout:28: seq_num 27 ranges {0=458752} 2023-10-20T03:35:16.299 INFO:tasks.rados.rados.0.smithi170.stdout:28: writing smithi17039300-28 from 0 to 458752 tid 1 2023-10-20T03:35:16.300 INFO:tasks.rados.rados.0.smithi170.stdout:16: finishing write tid 3 to smithi17039300-16 2023-10-20T03:35:16.300 INFO:tasks.rados.rados.0.smithi170.stdout:16: oid 16 updating version 0 to 2 2023-10-20T03:35:16.300 INFO:tasks.rados.rados.0.smithi170.stdout:16: oid 16 version 2 is already newer than 1 2023-10-20T03:35:16.300 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 16 v 2 (ObjNum 15 snap 0 seq_num 15) dirty exists 2023-10-20T03:35:16.300 INFO:tasks.rados.rados.0.smithi170.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2023-10-20T03:35:16.301 INFO:tasks.rados.rados.0.smithi170.stdout:14: finishing write tid 3 to smithi17039300-14 2023-10-20T03:35:16.301 INFO:tasks.rados.rados.0.smithi170.stdout:14: oid 14 updating version 0 to 4 2023-10-20T03:35:16.301 INFO:tasks.rados.rados.0.smithi170.stdout:14: oid 14 version 4 is already newer than 3 2023-10-20T03:35:16.301 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 4 (ObjNum 13 snap 0 seq_num 13) dirty exists 2023-10-20T03:35:16.301 INFO:tasks.rados.rados.0.smithi170.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2023-10-20T03:35:16.302 INFO:tasks.rados.rados.0.smithi170.stdout:17: finishing write tid 3 to smithi17039300-17 2023-10-20T03:35:16.302 INFO:tasks.rados.rados.0.smithi170.stdout:17: oid 17 updating version 0 to 6 2023-10-20T03:35:16.302 INFO:tasks.rados.rados.0.smithi170.stdout:17: oid 17 version 6 is already newer than 5 2023-10-20T03:35:16.302 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2023-10-20T03:35:16.302 INFO:tasks.rados.rados.0.smithi170.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2023-10-20T03:35:16.303 INFO:tasks.rados.rados.0.smithi170.stdout:14: done (14 left) 2023-10-20T03:35:16.303 INFO:tasks.rados.rados.0.smithi170.stdout:16: done (13 left) 2023-10-20T03:35:16.303 INFO:tasks.rados.rados.0.smithi170.stdout:17: done (12 left) 2023-10-20T03:35:16.303 INFO:tasks.rados.rados.0.smithi170.stdout:29: write initial oid 29 2023-10-20T03:35:16.303 INFO:tasks.rados.rados.0.smithi170.stdout:29: seq_num 28 ranges {0=475136} 2023-10-20T03:35:16.306 INFO:tasks.rados.rados.0.smithi170.stdout:29: writing smithi17039300-29 from 0 to 475136 tid 1 2023-10-20T03:35:16.306 INFO:tasks.rados.rados.0.smithi170.stdout:30: write initial oid 30 2023-10-20T03:35:16.307 INFO:tasks.rados.rados.0.smithi170.stdout:30: seq_num 29 ranges {0=491520} 2023-10-20T03:35:16.309 INFO:tasks.rados.rados.0.smithi170.stdout:30: writing smithi17039300-30 from 0 to 491520 tid 1 2023-10-20T03:35:16.309 INFO:tasks.rados.rados.0.smithi170.stdout:31: write initial oid 31 2023-10-20T03:35:16.310 INFO:tasks.rados.rados.0.smithi170.stdout:31: seq_num 30 ranges {0=507904} 2023-10-20T03:35:16.313 INFO:tasks.rados.rados.0.smithi170.stdout:31: writing smithi17039300-31 from 0 to 507904 tid 1 2023-10-20T03:35:16.313 INFO:tasks.rados.rados.0.smithi170.stdout:32: write initial oid 32 2023-10-20T03:35:16.313 INFO:tasks.rados.rados.0.smithi170.stdout:32: seq_num 31 ranges {0=524288} 2023-10-20T03:35:16.316 INFO:tasks.rados.rados.0.smithi170.stdout:32: writing smithi17039300-32 from 0 to 524288 tid 1 2023-10-20T03:35:16.317 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2023-10-20T03:35:16.319 INFO:tasks.rados.rados.0.smithi170.stdout:19: finishing write tid 1 to smithi17039300-19 2023-10-20T03:35:16.319 INFO:tasks.rados.rados.0.smithi170.stdout:19: finishing write tid 2 to smithi17039300-19 2023-10-20T03:35:16.319 INFO:tasks.rados.rados.0.smithi170.stdout:20: finishing write tid 1 to smithi17039300-20 2023-10-20T03:35:16.319 INFO:tasks.rados.rados.0.smithi170.stdout:20: finishing write tid 2 to smithi17039300-20 2023-10-20T03:35:16.320 INFO:tasks.rados.rados.0.smithi170.stdout:22: finishing write tid 1 to smithi17039300-22 2023-10-20T03:35:16.320 INFO:tasks.rados.rados.0.smithi170.stdout:22: finishing write tid 2 to smithi17039300-22 2023-10-20T03:35:16.320 INFO:tasks.rados.rados.0.smithi170.stdout:19: finishing write tid 3 to smithi17039300-19 2023-10-20T03:35:16.320 INFO:tasks.rados.rados.0.smithi170.stdout:19: oid 19 updating version 0 to 4 2023-10-20T03:35:16.320 INFO:tasks.rados.rados.0.smithi170.stdout:19: oid 19 version 4 is already newer than 3 2023-10-20T03:35:16.320 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 4 (ObjNum 18 snap 0 seq_num 18) dirty exists 2023-10-20T03:35:16.321 INFO:tasks.rados.rados.0.smithi170.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2023-10-20T03:35:16.321 INFO:tasks.rados.rados.0.smithi170.stdout:11: finishing write tid 3 to smithi17039300-11 2023-10-20T03:35:16.321 INFO:tasks.rados.rados.0.smithi170.stdout:11: oid 11 updating version 0 to 2 2023-10-20T03:35:16.321 INFO:tasks.rados.rados.0.smithi170.stdout:11: oid 11 version 2 is already newer than 1 2023-10-20T03:35:16.321 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 11 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2023-10-20T03:35:16.322 INFO:tasks.rados.rados.0.smithi170.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2023-10-20T03:35:16.322 INFO:tasks.rados.rados.0.smithi170.stdout:20: finishing write tid 3 to smithi17039300-20 2023-10-20T03:35:16.322 INFO:tasks.rados.rados.0.smithi170.stdout:20: oid 20 updating version 0 to 5 2023-10-20T03:35:16.322 INFO:tasks.rados.rados.0.smithi170.stdout:20: oid 20 updating version 5 to 6 2023-10-20T03:35:16.322 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2023-10-20T03:35:16.323 INFO:tasks.rados.rados.0.smithi170.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2023-10-20T03:35:16.323 INFO:tasks.rados.rados.0.smithi170.stdout:21: finishing write tid 1 to smithi17039300-21 2023-10-20T03:35:16.323 INFO:tasks.rados.rados.0.smithi170.stdout:21: finishing write tid 2 to smithi17039300-21 2023-10-20T03:35:16.323 INFO:tasks.rados.rados.0.smithi170.stdout:22: finishing write tid 3 to smithi17039300-22 2023-10-20T03:35:16.323 INFO:tasks.rados.rados.0.smithi170.stdout:22: oid 22 updating version 0 to 8 2023-10-20T03:35:16.324 INFO:tasks.rados.rados.0.smithi170.stdout:22: oid 22 version 8 is already newer than 7 2023-10-20T03:35:16.324 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 8 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-10-20T03:35:16.324 INFO:tasks.rados.rados.0.smithi170.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2023-10-20T03:35:16.324 INFO:tasks.rados.rados.0.smithi170.stdout:23: finishing write tid 1 to smithi17039300-23 2023-10-20T03:35:16.324 INFO:tasks.rados.rados.0.smithi170.stdout:8: finishing write tid 3 to smithi17039300-8 2023-10-20T03:35:16.325 INFO:tasks.rados.rados.0.smithi170.stdout:8: oid 8 updating version 0 to 1 2023-10-20T03:35:16.325 INFO:tasks.rados.rados.0.smithi170.stdout:8: oid 8 updating version 1 to 2 2023-10-20T03:35:16.325 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2023-10-20T03:35:16.325 INFO:tasks.rados.rados.0.smithi170.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2023-10-20T03:35:16.325 INFO:tasks.rados.rados.0.smithi170.stdout:24: finishing write tid 1 to smithi17039300-24 2023-10-20T03:35:16.325 INFO:tasks.rados.rados.0.smithi170.stdout:24: finishing write tid 2 to smithi17039300-24 2023-10-20T03:35:16.326 INFO:tasks.rados.rados.0.smithi170.stdout:8: done (15 left) 2023-10-20T03:35:16.326 INFO:tasks.rados.rados.0.smithi170.stdout:11: done (14 left) 2023-10-20T03:35:16.326 INFO:tasks.rados.rados.0.smithi170.stdout:19: done (13 left) 2023-10-20T03:35:16.326 INFO:tasks.rados.rados.0.smithi170.stdout:20: done (12 left) 2023-10-20T03:35:16.326 INFO:tasks.rados.rados.0.smithi170.stdout:22: done (11 left) 2023-10-20T03:35:16.327 INFO:tasks.rados.rados.0.smithi170.stdout:33: write initial oid 33 2023-10-20T03:35:16.327 INFO:tasks.rados.rados.0.smithi170.stdout:33: seq_num 32 ranges {0=540672} 2023-10-20T03:35:16.328 INFO:tasks.rados.rados.0.smithi170.stdout:33: writing smithi17039300-33 from 0 to 540672 tid 1 2023-10-20T03:35:16.328 INFO:tasks.rados.rados.0.smithi170.stdout:34: write initial oid 34 2023-10-20T03:35:16.328 INFO:tasks.rados.rados.0.smithi170.stdout:34: seq_num 33 ranges {0=557056} 2023-10-20T03:35:16.328 INFO:tasks.rados.rados.0.smithi170.stdout:34: writing smithi17039300-34 from 0 to 557056 tid 1 2023-10-20T03:35:16.328 INFO:tasks.rados.rados.0.smithi170.stdout:35: write initial oid 35 2023-10-20T03:35:16.329 INFO:tasks.rados.rados.0.smithi170.stdout:35: seq_num 34 ranges {0=573440} 2023-10-20T03:35:16.331 INFO:tasks.rados.rados.0.smithi170.stdout:35: writing smithi17039300-35 from 0 to 573440 tid 1 2023-10-20T03:35:16.331 INFO:tasks.rados.rados.0.smithi170.stdout:36: write initial oid 36 2023-10-20T03:35:16.331 INFO:tasks.rados.rados.0.smithi170.stdout:36: seq_num 35 ranges {0=589824} 2023-10-20T03:35:16.333 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:16.325+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:16.335 INFO:tasks.rados.rados.0.smithi170.stdout:36: writing smithi17039300-36 from 0 to 589824 tid 1 2023-10-20T03:35:16.336 INFO:tasks.rados.rados.0.smithi170.stdout:37: write initial oid 37 2023-10-20T03:35:16.336 INFO:tasks.rados.rados.0.smithi170.stdout:37: seq_num 36 ranges {0=606208} 2023-10-20T03:35:16.345 INFO:tasks.rados.rados.0.smithi170.stdout:37: writing smithi17039300-37 from 0 to 606208 tid 1 2023-10-20T03:35:16.346 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2023-10-20T03:35:16.346 INFO:tasks.rados.rados.0.smithi170.stdout:25: finishing write tid 1 to smithi17039300-25 2023-10-20T03:35:16.346 INFO:tasks.rados.rados.0.smithi170.stdout:26: finishing write tid 1 to smithi17039300-26 2023-10-20T03:35:16.346 INFO:tasks.rados.rados.0.smithi170.stdout:26: finishing write tid 2 to smithi17039300-26 2023-10-20T03:35:16.347 INFO:tasks.rados.rados.0.smithi170.stdout:21: finishing write tid 3 to smithi17039300-21 2023-10-20T03:35:16.347 INFO:tasks.rados.rados.0.smithi170.stdout:21: oid 21 updating version 0 to 3 2023-10-20T03:35:16.347 INFO:tasks.rados.rados.0.smithi170.stdout:21: oid 21 updating version 3 to 4 2023-10-20T03:35:16.347 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 21 v 4 (ObjNum 20 snap 0 seq_num 20) dirty exists 2023-10-20T03:35:16.347 INFO:tasks.rados.rados.0.smithi170.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2023-10-20T03:35:16.348 INFO:tasks.rados.rados.0.smithi170.stdout:29: finishing write tid 1 to smithi17039300-29 2023-10-20T03:35:16.348 INFO:tasks.rados.rados.0.smithi170.stdout:29: finishing write tid 2 to smithi17039300-29 2023-10-20T03:35:16.348 INFO:tasks.rados.rados.0.smithi170.stdout:25: finishing write tid 2 to smithi17039300-25 2023-10-20T03:35:16.348 INFO:tasks.rados.rados.0.smithi170.stdout:24: finishing write tid 3 to smithi17039300-24 2023-10-20T03:35:16.348 INFO:tasks.rados.rados.0.smithi170.stdout:24: oid 24 updating version 0 to 3 2023-10-20T03:35:16.348 INFO:tasks.rados.rados.0.smithi170.stdout:24: oid 24 updating version 3 to 4 2023-10-20T03:35:16.349 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 24 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2023-10-20T03:35:16.349 INFO:tasks.rados.rados.0.smithi170.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2023-10-20T03:35:16.349 INFO:tasks.rados.rados.0.smithi170.stdout:26: finishing write tid 3 to smithi17039300-26 2023-10-20T03:35:16.349 INFO:tasks.rados.rados.0.smithi170.stdout:26: oid 26 updating version 0 to 4 2023-10-20T03:35:16.349 INFO:tasks.rados.rados.0.smithi170.stdout:26: oid 26 version 4 is already newer than 3 2023-10-20T03:35:16.350 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 4 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-10-20T03:35:16.350 INFO:tasks.rados.rados.0.smithi170.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2023-10-20T03:35:16.350 INFO:tasks.rados.rados.0.smithi170.stdout:23: finishing write tid 2 to smithi17039300-23 2023-10-20T03:35:16.350 INFO:tasks.rados.rados.0.smithi170.stdout:28: finishing write tid 1 to smithi17039300-28 2023-10-20T03:35:16.350 INFO:tasks.rados.rados.0.smithi170.stdout:29: finishing write tid 3 to smithi17039300-29 2023-10-20T03:35:16.351 INFO:tasks.rados.rados.0.smithi170.stdout:29: oid 29 updating version 0 to 2 2023-10-20T03:35:16.351 INFO:tasks.rados.rados.0.smithi170.stdout:29: oid 29 version 2 is already newer than 1 2023-10-20T03:35:16.351 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 29 v 2 (ObjNum 28 snap 0 seq_num 28) dirty exists 2023-10-20T03:35:16.351 INFO:tasks.rados.rados.0.smithi170.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2023-10-20T03:35:16.351 INFO:tasks.rados.rados.0.smithi170.stdout:25: finishing write tid 3 to smithi17039300-25 2023-10-20T03:35:16.351 INFO:tasks.rados.rados.0.smithi170.stdout:25: oid 25 updating version 0 to 3 2023-10-20T03:35:16.352 INFO:tasks.rados.rados.0.smithi170.stdout:25: oid 25 updating version 3 to 4 2023-10-20T03:35:16.352 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 4 (ObjNum 24 snap 0 seq_num 24) dirty exists 2023-10-20T03:35:16.352 INFO:tasks.rados.rados.0.smithi170.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2023-10-20T03:35:16.352 INFO:tasks.rados.rados.0.smithi170.stdout:21: done (15 left) 2023-10-20T03:35:16.352 INFO:tasks.rados.rados.0.smithi170.stdout:24: done (14 left) 2023-10-20T03:35:16.353 INFO:tasks.rados.rados.0.smithi170.stdout:25: done (13 left) 2023-10-20T03:35:16.353 INFO:tasks.rados.rados.0.smithi170.stdout:26: done (12 left) 2023-10-20T03:35:16.353 INFO:tasks.rados.rados.0.smithi170.stdout:29: done (11 left) 2023-10-20T03:35:16.353 INFO:tasks.rados.rados.0.smithi170.stdout:38: write initial oid 38 2023-10-20T03:35:16.353 INFO:tasks.rados.rados.0.smithi170.stdout:38: seq_num 37 ranges {0=622592} 2023-10-20T03:35:16.354 INFO:tasks.rados.rados.0.smithi170.stdout:38: writing smithi17039300-38 from 0 to 622592 tid 1 2023-10-20T03:35:16.354 INFO:tasks.rados.rados.0.smithi170.stdout:39: write initial oid 39 2023-10-20T03:35:16.354 INFO:tasks.rados.rados.0.smithi170.stdout:39: seq_num 38 ranges {0=638976} 2023-10-20T03:35:16.355 INFO:tasks.rados.rados.0.smithi170.stdout:39: writing smithi17039300-39 from 0 to 638976 tid 1 2023-10-20T03:35:16.355 INFO:tasks.rados.rados.0.smithi170.stdout:40: write initial oid 40 2023-10-20T03:35:16.355 INFO:tasks.rados.rados.0.smithi170.stdout:40: seq_num 39 ranges {0=671744} 2023-10-20T03:35:16.358 INFO:tasks.rados.rados.0.smithi170.stdout:40: writing smithi17039300-40 from 0 to 671744 tid 1 2023-10-20T03:35:16.358 INFO:tasks.rados.rados.0.smithi170.stdout:41: write initial oid 41 2023-10-20T03:35:16.359 INFO:tasks.rados.rados.0.smithi170.stdout:41: seq_num 40 ranges {0=688128} 2023-10-20T03:35:16.363 INFO:tasks.rados.rados.0.smithi170.stdout:41: writing smithi17039300-41 from 0 to 688128 tid 1 2023-10-20T03:35:16.363 INFO:tasks.rados.rados.0.smithi170.stdout:42: write initial oid 42 2023-10-20T03:35:16.363 INFO:tasks.rados.rados.0.smithi170.stdout:42: seq_num 41 ranges {0=704512} 2023-10-20T03:35:16.369 INFO:tasks.rados.rados.0.smithi170.stdout:42: writing smithi17039300-42 from 0 to 704512 tid 1 2023-10-20T03:35:16.370 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2023-10-20T03:35:16.370 INFO:tasks.rados.rados.0.smithi170.stdout:32: finishing write tid 1 to smithi17039300-32 2023-10-20T03:35:16.370 INFO:tasks.rados.rados.0.smithi170.stdout:32: finishing write tid 2 to smithi17039300-32 2023-10-20T03:35:16.370 INFO:tasks.rados.rados.0.smithi170.stdout:28: finishing write tid 2 to smithi17039300-28 2023-10-20T03:35:16.370 INFO:tasks.rados.rados.0.smithi170.stdout:27: finishing write tid 1 to smithi17039300-27 2023-10-20T03:35:16.371 INFO:tasks.rados.rados.0.smithi170.stdout:27: finishing write tid 2 to smithi17039300-27 2023-10-20T03:35:16.371 INFO:tasks.rados.rados.0.smithi170.stdout:30: finishing write tid 1 to smithi17039300-30 2023-10-20T03:35:16.371 INFO:tasks.rados.rados.0.smithi170.stdout:30: finishing write tid 2 to smithi17039300-30 2023-10-20T03:35:16.371 INFO:tasks.rados.rados.0.smithi170.stdout:23: finishing write tid 3 to smithi17039300-23 2023-10-20T03:35:16.371 INFO:tasks.rados.rados.0.smithi170.stdout:23: oid 23 updating version 0 to 3 2023-10-20T03:35:16.372 INFO:tasks.rados.rados.0.smithi170.stdout:23: oid 23 updating version 3 to 4 2023-10-20T03:35:16.372 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 23 v 4 (ObjNum 22 snap 0 seq_num 22) dirty exists 2023-10-20T03:35:16.372 INFO:tasks.rados.rados.0.smithi170.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2023-10-20T03:35:16.372 INFO:tasks.rados.rados.0.smithi170.stdout:31: finishing write tid 1 to smithi17039300-31 2023-10-20T03:35:16.372 INFO:tasks.rados.rados.0.smithi170.stdout:31: finishing write tid 2 to smithi17039300-31 2023-10-20T03:35:16.372 INFO:tasks.rados.rados.0.smithi170.stdout:23: done (15 left) 2023-10-20T03:35:16.373 INFO:tasks.rados.rados.0.smithi170.stdout:43: write initial oid 43 2023-10-20T03:35:16.373 INFO:tasks.rados.rados.0.smithi170.stdout:43: seq_num 42 ranges {0=720896} 2023-10-20T03:35:16.374 INFO:tasks.rados.rados.0.smithi170.stdout:43: writing smithi17039300-43 from 0 to 720896 tid 1 2023-10-20T03:35:16.374 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2023-10-20T03:35:16.374 INFO:tasks.rados.rados.0.smithi170.stdout:33: finishing write tid 1 to smithi17039300-33 2023-10-20T03:35:16.375 INFO:tasks.rados.rados.0.smithi170.stdout:27: finishing write tid 3 to smithi17039300-27 2023-10-20T03:35:16.375 INFO:tasks.rados.rados.0.smithi170.stdout:27: oid 27 updating version 0 to 6 2023-10-20T03:35:16.375 INFO:tasks.rados.rados.0.smithi170.stdout:27: oid 27 version 6 is already newer than 5 2023-10-20T03:35:16.375 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2023-10-20T03:35:16.376 INFO:tasks.rados.rados.0.smithi170.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2023-10-20T03:35:16.376 INFO:tasks.rados.rados.0.smithi170.stdout:30: finishing write tid 3 to smithi17039300-30 2023-10-20T03:35:16.376 INFO:tasks.rados.rados.0.smithi170.stdout:30: oid 30 updating version 0 to 7 2023-10-20T03:35:16.376 INFO:tasks.rados.rados.0.smithi170.stdout:30: oid 30 updating version 7 to 8 2023-10-20T03:35:16.376 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 8 (ObjNum 29 snap 0 seq_num 29) dirty exists 2023-10-20T03:35:16.376 INFO:tasks.rados.rados.0.smithi170.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2023-10-20T03:35:16.377 INFO:tasks.rados.rados.0.smithi170.stdout:33: finishing write tid 2 to smithi17039300-33 2023-10-20T03:35:16.377 INFO:tasks.rados.rados.0.smithi170.stdout:35: finishing write tid 1 to smithi17039300-35 2023-10-20T03:35:16.377 INFO:tasks.rados.rados.0.smithi170.stdout:27: done (15 left) 2023-10-20T03:35:16.377 INFO:tasks.rados.rados.0.smithi170.stdout:30: done (14 left) 2023-10-20T03:35:16.377 INFO:tasks.rados.rados.0.smithi170.stdout:44: write initial oid 44 2023-10-20T03:35:16.378 INFO:tasks.rados.rados.0.smithi170.stdout:44: seq_num 43 ranges {0=737280} 2023-10-20T03:35:16.378 INFO:tasks.rados.rados.0.smithi170.stdout:44: writing smithi17039300-44 from 0 to 737280 tid 1 2023-10-20T03:35:16.379 INFO:tasks.rados.rados.0.smithi170.stdout:45: write initial oid 45 2023-10-20T03:35:16.379 INFO:tasks.rados.rados.0.smithi170.stdout:45: seq_num 44 ranges {0=753664} 2023-10-20T03:35:16.384 INFO:tasks.rados.rados.0.smithi170.stdout:45: writing smithi17039300-45 from 0 to 753664 tid 1 2023-10-20T03:35:16.384 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 16 2023-10-20T03:35:16.384 INFO:tasks.rados.rados.0.smithi170.stdout:28: finishing write tid 3 to smithi17039300-28 2023-10-20T03:35:16.384 INFO:tasks.rados.rados.0.smithi170.stdout:28: oid 28 updating version 0 to 6 2023-10-20T03:35:16.384 INFO:tasks.rados.rados.0.smithi170.stdout:28: oid 28 version 6 is already newer than 5 2023-10-20T03:35:16.385 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2023-10-20T03:35:16.385 INFO:tasks.rados.rados.0.smithi170.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2023-10-20T03:35:16.385 INFO:tasks.rados.rados.0.smithi170.stdout:35: finishing write tid 2 to smithi17039300-35 2023-10-20T03:35:16.385 INFO:tasks.rados.rados.0.smithi170.stdout:32: finishing write tid 3 to smithi17039300-32 2023-10-20T03:35:16.386 INFO:tasks.rados.rados.0.smithi170.stdout:32: oid 32 updating version 0 to 2 2023-10-20T03:35:16.386 INFO:tasks.rados.rados.0.smithi170.stdout:32: oid 32 version 2 is already newer than 1 2023-10-20T03:35:16.386 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 2 (ObjNum 31 snap 0 seq_num 31) dirty exists 2023-10-20T03:35:16.386 INFO:tasks.rados.rados.0.smithi170.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2023-10-20T03:35:16.386 INFO:tasks.rados.rados.0.smithi170.stdout:34: finishing write tid 1 to smithi17039300-34 2023-10-20T03:35:16.387 INFO:tasks.rados.rados.0.smithi170.stdout:31: finishing write tid 3 to smithi17039300-31 2023-10-20T03:35:16.387 INFO:tasks.rados.rados.0.smithi170.stdout:31: oid 31 updating version 0 to 8 2023-10-20T03:35:16.387 INFO:tasks.rados.rados.0.smithi170.stdout:31: oid 31 version 8 is already newer than 7 2023-10-20T03:35:16.387 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 8 (ObjNum 30 snap 0 seq_num 30) dirty exists 2023-10-20T03:35:16.387 INFO:tasks.rados.rados.0.smithi170.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2023-10-20T03:35:16.388 INFO:tasks.rados.rados.0.smithi170.stdout:34: finishing write tid 2 to smithi17039300-34 2023-10-20T03:35:16.388 INFO:tasks.rados.rados.0.smithi170.stdout:36: finishing write tid 1 to smithi17039300-36 2023-10-20T03:35:16.388 INFO:tasks.rados.rados.0.smithi170.stdout:35: finishing write tid 3 to smithi17039300-35 2023-10-20T03:35:16.388 INFO:tasks.rados.rados.0.smithi170.stdout:35: oid 35 updating version 0 to 4 2023-10-20T03:35:16.388 INFO:tasks.rados.rados.0.smithi170.stdout:35: oid 35 version 4 is already newer than 3 2023-10-20T03:35:16.389 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 4 (ObjNum 34 snap 0 seq_num 34) dirty exists 2023-10-20T03:35:16.389 INFO:tasks.rados.rados.0.smithi170.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2023-10-20T03:35:16.389 INFO:tasks.rados.rados.0.smithi170.stdout:36: finishing write tid 2 to smithi17039300-36 2023-10-20T03:35:16.389 INFO:tasks.rados.rados.0.smithi170.stdout:34: finishing write tid 3 to smithi17039300-34 2023-10-20T03:35:16.389 INFO:tasks.rados.rados.0.smithi170.stdout:34: oid 34 updating version 0 to 5 2023-10-20T03:35:16.389 INFO:tasks.rados.rados.0.smithi170.stdout:34: oid 34 updating version 5 to 6 2023-10-20T03:35:16.390 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2023-10-20T03:35:16.390 INFO:tasks.rados.rados.0.smithi170.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2023-10-20T03:35:16.390 INFO:tasks.rados.rados.0.smithi170.stdout:33: finishing write tid 3 to smithi17039300-33 2023-10-20T03:35:16.390 INFO:tasks.rados.rados.0.smithi170.stdout:33: oid 33 updating version 0 to 10 2023-10-20T03:35:16.390 INFO:tasks.rados.rados.0.smithi170.stdout:33: oid 33 version 10 is already newer than 9 2023-10-20T03:35:16.391 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 33 v 10 (ObjNum 32 snap 0 seq_num 32) dirty exists 2023-10-20T03:35:16.391 INFO:tasks.rados.rados.0.smithi170.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2023-10-20T03:35:16.391 INFO:tasks.rados.rados.0.smithi170.stdout:36: finishing write tid 3 to smithi17039300-36 2023-10-20T03:35:16.391 INFO:tasks.rados.rados.0.smithi170.stdout:36: oid 36 updating version 0 to 6 2023-10-20T03:35:16.391 INFO:tasks.rados.rados.0.smithi170.stdout:36: oid 36 version 6 is already newer than 5 2023-10-20T03:35:16.391 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-10-20T03:35:16.392 INFO:tasks.rados.rados.0.smithi170.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2023-10-20T03:35:16.392 INFO:tasks.rados.rados.0.smithi170.stdout:37: finishing write tid 1 to smithi17039300-37 2023-10-20T03:35:16.392 INFO:tasks.rados.rados.0.smithi170.stdout:37: finishing write tid 2 to smithi17039300-37 2023-10-20T03:35:16.392 INFO:tasks.rados.rados.0.smithi170.stdout:38: finishing write tid 1 to smithi17039300-38 2023-10-20T03:35:16.392 INFO:tasks.rados.rados.0.smithi170.stdout:38: finishing write tid 2 to smithi17039300-38 2023-10-20T03:35:16.393 INFO:tasks.rados.rados.0.smithi170.stdout:38: finishing write tid 3 to smithi17039300-38 2023-10-20T03:35:16.393 INFO:tasks.rados.rados.0.smithi170.stdout:38: oid 38 updating version 0 to 3 2023-10-20T03:35:16.393 INFO:tasks.rados.rados.0.smithi170.stdout:38: oid 38 updating version 3 to 4 2023-10-20T03:35:16.393 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 38 v 4 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-10-20T03:35:16.393 INFO:tasks.rados.rados.0.smithi170.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2023-10-20T03:35:16.393 INFO:tasks.rados.rados.0.smithi170.stdout:37: finishing write tid 3 to smithi17039300-37 2023-10-20T03:35:16.394 INFO:tasks.rados.rados.0.smithi170.stdout:37: oid 37 updating version 0 to 5 2023-10-20T03:35:16.394 INFO:tasks.rados.rados.0.smithi170.stdout:37: oid 37 updating version 5 to 6 2023-10-20T03:35:16.394 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2023-10-20T03:35:16.394 INFO:tasks.rados.rados.0.smithi170.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2023-10-20T03:35:16.394 INFO:tasks.rados.rados.0.smithi170.stdout:40: finishing write tid 1 to smithi17039300-40 2023-10-20T03:35:16.395 INFO:tasks.rados.rados.0.smithi170.stdout:40: finishing write tid 2 to smithi17039300-40 2023-10-20T03:35:16.395 INFO:tasks.rados.rados.0.smithi170.stdout:28: done (15 left) 2023-10-20T03:35:16.395 INFO:tasks.rados.rados.0.smithi170.stdout:31: done (14 left) 2023-10-20T03:35:16.395 INFO:tasks.rados.rados.0.smithi170.stdout:32: done (13 left) 2023-10-20T03:35:16.396 INFO:tasks.rados.rados.0.smithi170.stdout:33: done (12 left) 2023-10-20T03:35:16.396 INFO:tasks.rados.rados.0.smithi170.stdout:34: done (11 left) 2023-10-20T03:35:16.396 INFO:tasks.rados.rados.0.smithi170.stdout:35: done (10 left) 2023-10-20T03:35:16.396 INFO:tasks.rados.rados.0.smithi170.stdout:36: done (9 left) 2023-10-20T03:35:16.396 INFO:tasks.rados.rados.0.smithi170.stdout:37: done (8 left) 2023-10-20T03:35:16.397 INFO:tasks.rados.rados.0.smithi170.stdout:38: done (7 left) 2023-10-20T03:35:16.397 INFO:tasks.rados.rados.0.smithi170.stdout:46: write initial oid 46 2023-10-20T03:35:16.397 INFO:tasks.rados.rados.0.smithi170.stdout:46: seq_num 45 ranges {0=770048} 2023-10-20T03:35:16.398 INFO:tasks.rados.rados.0.smithi170.stdout:46: writing smithi17039300-46 from 0 to 770048 tid 1 2023-10-20T03:35:16.398 INFO:tasks.rados.rados.0.smithi170.stdout:47: write initial oid 47 2023-10-20T03:35:16.398 INFO:tasks.rados.rados.0.smithi170.stdout:47: seq_num 46 ranges {0=786432} 2023-10-20T03:35:16.398 INFO:tasks.rados.rados.0.smithi170.stdout:47: writing smithi17039300-47 from 0 to 786432 tid 1 2023-10-20T03:35:16.399 INFO:tasks.rados.rados.0.smithi170.stdout:48: write initial oid 48 2023-10-20T03:35:16.399 INFO:tasks.rados.rados.0.smithi170.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2023-10-20T03:35:16.403 INFO:tasks.rados.rados.0.smithi170.stdout:48: writing smithi17039300-48 from 0 to 425984 tid 1 2023-10-20T03:35:16.403 INFO:tasks.rados.rados.0.smithi170.stdout:48: writing smithi17039300-48 from 425984 to 802816 tid 2 2023-10-20T03:35:16.403 INFO:tasks.rados.rados.0.smithi170.stdout:49: write initial oid 49 2023-10-20T03:35:16.403 INFO:tasks.rados.rados.0.smithi170.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2023-10-20T03:35:16.404 INFO:tasks.rados.rados.0.smithi170.stdout:49: writing smithi17039300-49 from 0 to 442368 tid 1 2023-10-20T03:35:16.404 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:16.405 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:16.405 INFO:tasks.rados.rados.0.smithi170.stdout:49: writing smithi17039300-49 from 442368 to 819200 tid 2 2023-10-20T03:35:16.405 INFO:tasks.rados.rados.0.smithi170.stdout:50: write initial oid 50 2023-10-20T03:35:16.406 INFO:tasks.rados.rados.0.smithi170.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2023-10-20T03:35:16.408 INFO:tasks.rados.rados.0.smithi170.stdout:50: writing smithi17039300-50 from 0 to 458752 tid 1 2023-10-20T03:35:16.414 INFO:tasks.rados.rados.0.smithi170.stdout:50: writing smithi17039300-50 from 458752 to 835584 tid 2 2023-10-20T03:35:16.415 INFO:tasks.rados.rados.0.smithi170.stdout:51: rollback oid 29 current snap is 0 2023-10-20T03:35:16.415 INFO:tasks.rados.rados.0.smithi170.stdout:51: done (12 left) 2023-10-20T03:35:16.415 INFO:tasks.rados.rados.0.smithi170.stdout:52: delete oid 7 current snap is 0 2023-10-20T03:35:16.415 INFO:tasks.rados.rados.0.smithi170.stdout:39: finishing write tid 1 to smithi17039300-39 2023-10-20T03:35:16.415 INFO:tasks.rados.rados.0.smithi170.stdout:39: finishing write tid 2 to smithi17039300-39 2023-10-20T03:35:16.416 INFO:tasks.rados.rados.0.smithi170.stdout:40: finishing write tid 3 to smithi17039300-40 2023-10-20T03:35:16.416 INFO:tasks.rados.rados.0.smithi170.stdout:40: oid 40 updating version 0 to 7 2023-10-20T03:35:16.416 INFO:tasks.rados.rados.0.smithi170.stdout:40: oid 40 updating version 7 to 8 2023-10-20T03:35:16.416 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 8 (ObjNum 39 snap 0 seq_num 39) dirty exists 2023-10-20T03:35:16.417 INFO:tasks.rados.rados.0.smithi170.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2023-10-20T03:35:16.417 INFO:tasks.rados.rados.0.smithi170.stdout:39: finishing write tid 3 to smithi17039300-39 2023-10-20T03:35:16.417 INFO:tasks.rados.rados.0.smithi170.stdout:39: oid 39 updating version 0 to 4 2023-10-20T03:35:16.417 INFO:tasks.rados.rados.0.smithi170.stdout:39: oid 39 version 4 is already newer than 3 2023-10-20T03:35:16.417 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 4 (ObjNum 38 snap 0 seq_num 38) dirty exists 2023-10-20T03:35:16.418 INFO:tasks.rados.rados.0.smithi170.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2023-10-20T03:35:16.418 INFO:tasks.rados.rados.0.smithi170.stdout:43: finishing write tid 1 to smithi17039300-43 2023-10-20T03:35:16.418 INFO:tasks.rados.rados.0.smithi170.stdout:42: finishing write tid 1 to smithi17039300-42 2023-10-20T03:35:16.418 INFO:tasks.rados.rados.0.smithi170.stdout:42: finishing write tid 2 to smithi17039300-42 2023-10-20T03:35:16.419 INFO:tasks.rados.rados.0.smithi170.stdout:41: finishing write tid 1 to smithi17039300-41 2023-10-20T03:35:16.419 INFO:tasks.rados.rados.0.smithi170.stdout:41: finishing write tid 2 to smithi17039300-41 2023-10-20T03:35:16.419 INFO:tasks.rados.rados.0.smithi170.stdout:44: finishing write tid 1 to smithi17039300-44 2023-10-20T03:35:16.419 INFO:tasks.rados.rados.0.smithi170.stdout:44: finishing write tid 2 to smithi17039300-44 2023-10-20T03:35:16.419 INFO:tasks.rados.rados.0.smithi170.stdout:41: finishing write tid 3 to smithi17039300-41 2023-10-20T03:35:16.420 INFO:tasks.rados.rados.0.smithi170.stdout:41: oid 41 updating version 0 to 6 2023-10-20T03:35:16.420 INFO:tasks.rados.rados.0.smithi170.stdout:41: oid 41 version 6 is already newer than 5 2023-10-20T03:35:16.420 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2023-10-20T03:35:16.420 INFO:tasks.rados.rados.0.smithi170.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2023-10-20T03:35:16.420 INFO:tasks.rados.rados.0.smithi170.stdout:42: finishing write tid 3 to smithi17039300-42 2023-10-20T03:35:16.421 INFO:tasks.rados.rados.0.smithi170.stdout:42: oid 42 updating version 0 to 6 2023-10-20T03:35:16.421 INFO:tasks.rados.rados.0.smithi170.stdout:42: oid 42 version 6 is already newer than 5 2023-10-20T03:35:16.421 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2023-10-20T03:35:16.421 INFO:tasks.rados.rados.0.smithi170.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2023-10-20T03:35:16.421 INFO:tasks.rados.rados.0.smithi170.stdout:44: finishing write tid 3 to smithi17039300-44 2023-10-20T03:35:16.422 INFO:tasks.rados.rados.0.smithi170.stdout:44: oid 44 updating version 0 to 8 2023-10-20T03:35:16.422 INFO:tasks.rados.rados.0.smithi170.stdout:44: oid 44 version 8 is already newer than 7 2023-10-20T03:35:16.422 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 8 (ObjNum 43 snap 0 seq_num 43) dirty exists 2023-10-20T03:35:16.422 INFO:tasks.rados.rados.0.smithi170.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2023-10-20T03:35:16.422 INFO:tasks.rados.rados.0.smithi170.stdout:43: finishing write tid 2 to smithi17039300-43 2023-10-20T03:35:16.423 INFO:tasks.rados.rados.0.smithi170.stdout:45: finishing write tid 1 to smithi17039300-45 2023-10-20T03:35:16.423 INFO:tasks.rados.rados.0.smithi170.stdout:43: finishing write tid 3 to smithi17039300-43 2023-10-20T03:35:16.423 INFO:tasks.rados.rados.0.smithi170.stdout:43: oid 43 updating version 0 to 7 2023-10-20T03:35:16.423 INFO:tasks.rados.rados.0.smithi170.stdout:43: oid 43 updating version 7 to 8 2023-10-20T03:35:16.424 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 43 v 8 (ObjNum 42 snap 0 seq_num 42) dirty exists 2023-10-20T03:35:16.424 INFO:tasks.rados.rados.0.smithi170.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2023-10-20T03:35:16.424 INFO:tasks.rados.rados.0.smithi170.stdout:45: finishing write tid 2 to smithi17039300-45 2023-10-20T03:35:16.424 INFO:tasks.rados.rados.0.smithi170.stdout:48: finishing write tid 1 to smithi17039300-48 2023-10-20T03:35:16.424 INFO:tasks.rados.rados.0.smithi170.stdout:48: finishing write tid 2 to smithi17039300-48 2023-10-20T03:35:16.425 INFO:tasks.rados.rados.0.smithi170.stdout:45: finishing write tid 3 to smithi17039300-45 2023-10-20T03:35:16.425 INFO:tasks.rados.rados.0.smithi170.stdout:45: oid 45 updating version 0 to 4 2023-10-20T03:35:16.425 INFO:tasks.rados.rados.0.smithi170.stdout:45: oid 45 version 4 is already newer than 3 2023-10-20T03:35:16.425 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 4 (ObjNum 44 snap 0 seq_num 44) dirty exists 2023-10-20T03:35:16.425 INFO:tasks.rados.rados.0.smithi170.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2023-10-20T03:35:16.425 INFO:tasks.rados.rados.0.smithi170.stdout:48: finishing write tid 3 to smithi17039300-48 2023-10-20T03:35:16.428 INFO:tasks.rados.rados.0.smithi170.stdout:47: finishing write tid 1 to smithi17039300-47 2023-10-20T03:35:16.428 INFO:tasks.rados.rados.0.smithi170.stdout:47: finishing write tid 2 to smithi17039300-47 2023-10-20T03:35:16.428 INFO:tasks.rados.rados.0.smithi170.stdout:46: finishing write tid 1 to smithi17039300-46 2023-10-20T03:35:16.429 INFO:tasks.rados.rados.0.smithi170.stdout:46: finishing write tid 2 to smithi17039300-46 2023-10-20T03:35:16.429 INFO:tasks.rados.rados.0.smithi170.stdout:49: finishing write tid 1 to smithi17039300-49 2023-10-20T03:35:16.429 INFO:tasks.rados.rados.0.smithi170.stdout:48: finishing write tid 4 to smithi17039300-48 2023-10-20T03:35:16.429 INFO:tasks.rados.rados.0.smithi170.stdout:48: oid 48 updating version 0 to 3 2023-10-20T03:35:16.430 INFO:tasks.rados.rados.0.smithi170.stdout:48: oid 48 updating version 3 to 4 2023-10-20T03:35:16.430 INFO:tasks.rados.rados.0.smithi170.stdout:48: oid 48 updating version 4 to 5 2023-10-20T03:35:16.430 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 48 v 5 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-10-20T03:35:16.431 INFO:tasks.rados.rados.0.smithi170.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2023-10-20T03:35:16.431 INFO:tasks.rados.rados.0.smithi170.stdout:50: finishing write tid 1 to smithi17039300-50 2023-10-20T03:35:16.431 INFO:tasks.rados.rados.0.smithi170.stdout:39: done (12 left) 2023-10-20T03:35:16.432 INFO:tasks.rados.rados.0.smithi170.stdout:40: done (11 left) 2023-10-20T03:35:16.432 INFO:tasks.rados.rados.0.smithi170.stdout:41: done (10 left) 2023-10-20T03:35:16.432 INFO:tasks.rados.rados.0.smithi170.stdout:42: done (9 left) 2023-10-20T03:35:16.433 INFO:tasks.rados.rados.0.smithi170.stdout:43: done (8 left) 2023-10-20T03:35:16.433 INFO:tasks.rados.rados.0.smithi170.stdout:44: done (7 left) 2023-10-20T03:35:16.433 INFO:tasks.rados.rados.0.smithi170.stdout:45: done (6 left) 2023-10-20T03:35:16.434 INFO:tasks.rados.rados.0.smithi170.stdout:48: done (5 left) 2023-10-20T03:35:16.434 INFO:tasks.rados.rados.0.smithi170.stdout:52: done (4 left) 2023-10-20T03:35:16.434 INFO:tasks.rados.rados.0.smithi170.stdout:53: rollback oid 43 current snap is 0 2023-10-20T03:35:16.435 INFO:tasks.rados.rados.0.smithi170.stdout:53: done (4 left) 2023-10-20T03:35:16.435 INFO:tasks.rados.rados.0.smithi170.stdout:54: delete oid 33 current snap is 0 2023-10-20T03:35:16.435 INFO:tasks.rados.rados.0.smithi170.stdout:49: finishing write tid 2 to smithi17039300-49 2023-10-20T03:35:16.435 INFO:tasks.rados.rados.0.smithi170.stdout:47: finishing write tid 3 to smithi17039300-47 2023-10-20T03:35:16.436 INFO:tasks.rados.rados.0.smithi170.stdout:47: oid 47 updating version 0 to 7 2023-10-20T03:35:16.436 INFO:tasks.rados.rados.0.smithi170.stdout:47: oid 47 updating version 7 to 8 2023-10-20T03:35:16.436 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 47 v 8 (ObjNum 46 snap 0 seq_num 46) dirty exists 2023-10-20T03:35:16.436 INFO:tasks.rados.rados.0.smithi170.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2023-10-20T03:35:16.437 INFO:tasks.rados.rados.0.smithi170.stdout:49: finishing write tid 3 to smithi17039300-49 2023-10-20T03:35:16.437 INFO:tasks.rados.rados.0.smithi170.stdout:50: finishing write tid 2 to smithi17039300-50 2023-10-20T03:35:16.437 INFO:tasks.rados.rados.0.smithi170.stdout:47: done (4 left) 2023-10-20T03:35:16.437 INFO:tasks.rados.rados.0.smithi170.stdout:54: done (3 left) 2023-10-20T03:35:16.438 INFO:tasks.rados.rados.0.smithi170.stdout:55: copy_from oid 10 from oid 7 current snap is 0 2023-10-20T03:35:16.438 INFO:tasks.rados.rados.0.smithi170.stdout:56: rollback oid 48 current snap is 0 2023-10-20T03:35:16.438 INFO:tasks.rados.rados.0.smithi170.stdout:56: done (4 left) 2023-10-20T03:35:16.438 INFO:tasks.rados.rados.0.smithi170.stdout:57: read oid 21 snap -1 2023-10-20T03:35:16.438 INFO:tasks.rados.rados.0.smithi170.stdout:57: expect (ObjNum 20 snap 0 seq_num 20) 2023-10-20T03:35:16.439 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 33 current snap is 0 2023-10-20T03:35:16.439 INFO:tasks.rados.rados.0.smithi170.stdout:58: seq_num 52 ranges {0=507904,507904=376832} 2023-10-20T03:35:16.439 INFO:tasks.rados.rados.0.smithi170.stdout:58: writing smithi17039300-33 from 0 to 507904 tid 1 2023-10-20T03:35:16.441 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-10-20T03:35:16.442 INFO:tasks.rados.rados.0.smithi170.stdout:58: writing smithi17039300-33 from 507904 to 884736 tid 2 2023-10-20T03:35:16.442 INFO:tasks.rados.rados.0.smithi170.stdout:59: delete oid 19 current snap is 0 2023-10-20T03:35:16.443 INFO:tasks.rados.rados.0.smithi170.stdout:46: finishing write tid 3 to smithi17039300-46 2023-10-20T03:35:16.443 INFO:tasks.rados.rados.0.smithi170.stdout:46: oid 46 updating version 0 to 4 2023-10-20T03:35:16.443 INFO:tasks.rados.rados.0.smithi170.stdout:46: oid 46 version 4 is already newer than 3 2023-10-20T03:35:16.443 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 46 v 4 (ObjNum 45 snap 0 seq_num 45) dirty exists 2023-10-20T03:35:16.443 INFO:tasks.rados.rados.0.smithi170.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2023-10-20T03:35:16.444 INFO:tasks.rados.rados.0.smithi170.stdout:49: finishing write tid 4 to smithi17039300-49 2023-10-20T03:35:16.444 INFO:tasks.rados.rados.0.smithi170.stdout:49: oid 49 updating version 0 to 7 2023-10-20T03:35:16.444 INFO:tasks.rados.rados.0.smithi170.stdout:49: oid 49 version 7 is already newer than 5 2023-10-20T03:35:16.444 INFO:tasks.rados.rados.0.smithi170.stdout:49: oid 49 version 7 is already newer than 6 2023-10-20T03:35:16.444 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 49 v 7 (ObjNum 48 snap 0 seq_num 48) dirty exists 2023-10-20T03:35:16.445 INFO:tasks.rados.rados.0.smithi170.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2023-10-20T03:35:16.445 INFO:tasks.rados.rados.0.smithi170.stdout:50: finishing write tid 3 to smithi17039300-50 2023-10-20T03:35:16.445 INFO:tasks.rados.rados.0.smithi170.stdout:55: finishing copy_from to smithi17039300-10 2023-10-20T03:35:16.445 INFO:tasks.rados.rados.0.smithi170.stdout:55: got expected ENOENT (src dne) 2023-10-20T03:35:16.445 INFO:tasks.rados.rados.0.smithi170.stdout:55: finishing copy_from racing read to smithi17039300-10 2023-10-20T03:35:16.446 INFO:tasks.rados.rados.0.smithi170.stdout:57: expect (ObjNum 20 snap 0 seq_num 20) 2023-10-20T03:35:16.446 INFO:tasks.rados.rados.0.smithi170.stdout:50: finishing write tid 4 to smithi17039300-50 2023-10-20T03:35:16.446 INFO:tasks.rados.rados.0.smithi170.stdout:50: oid 50 updating version 0 to 8 2023-10-20T03:35:16.446 INFO:tasks.rados.rados.0.smithi170.stdout:50: oid 50 version 8 is already newer than 7 2023-10-20T03:35:16.446 INFO:tasks.rados.rados.0.smithi170.stdout:50: oid 50 updating version 8 to 9 2023-10-20T03:35:16.447 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2023-10-20T03:35:16.447 INFO:tasks.rados.rados.0.smithi170.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2023-10-20T03:35:16.447 INFO:tasks.rados.rados.0.smithi170.stdout:58: finishing write tid 1 to smithi17039300-33 2023-10-20T03:35:16.447 INFO:tasks.rados.rados.0.smithi170.stdout:58: finishing write tid 2 to smithi17039300-33 2023-10-20T03:35:16.448 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:16.439+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:16.449 INFO:tasks.rados.rados.0.smithi170.stdout:46: done (6 left) 2023-10-20T03:35:16.450 INFO:tasks.rados.rados.0.smithi170.stdout:49: done (5 left) 2023-10-20T03:35:16.450 INFO:tasks.rados.rados.0.smithi170.stdout:50: done (4 left) 2023-10-20T03:35:16.450 INFO:tasks.rados.rados.0.smithi170.stdout:55: done (3 left) 2023-10-20T03:35:16.450 INFO:tasks.rados.rados.0.smithi170.stdout:57: done (2 left) 2023-10-20T03:35:16.450 INFO:tasks.rados.rados.0.smithi170.stdout:59: done (1 left) 2023-10-20T03:35:16.451 INFO:tasks.rados.rados.0.smithi170.stdout:60: delete oid 38 current snap is 0 2023-10-20T03:35:16.451 INFO:tasks.rados.rados.0.smithi170.stdout:58: finishing write tid 3 to smithi17039300-33 2023-10-20T03:35:16.451 INFO:tasks.rados.rados.0.smithi170.stdout:58: finishing write tid 4 to smithi17039300-33 2023-10-20T03:35:16.451 INFO:tasks.rados.rados.0.smithi170.stdout:58: oid 33 updating version 0 to 14 2023-10-20T03:35:16.451 INFO:tasks.rados.rados.0.smithi170.stdout:58: oid 33 version 14 is already newer than 12 2023-10-20T03:35:16.452 INFO:tasks.rados.rados.0.smithi170.stdout:58: oid 33 version 14 is already newer than 13 2023-10-20T03:35:16.452 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 33 v 14 (ObjNum 52 snap 0 seq_num 52) dirty exists 2023-10-20T03:35:16.452 INFO:tasks.rados.rados.0.smithi170.stdout:58: left oid 33 (ObjNum 52 snap 0 seq_num 52) 2023-10-20T03:35:16.452 INFO:tasks.rados.rados.0.smithi170.stdout:58: done (1 left) 2023-10-20T03:35:16.452 INFO:tasks.rados.rados.0.smithi170.stdout:60: done (0 left) 2023-10-20T03:35:16.453 INFO:tasks.rados.rados.0.smithi170.stdout:61: copy_from oid 29 from oid 26 current snap is 0 2023-10-20T03:35:16.453 INFO:tasks.rados.rados.0.smithi170.stdout:62: read oid 46 snap -1 2023-10-20T03:35:16.453 INFO:tasks.rados.rados.0.smithi170.stdout:62: expect (ObjNum 45 snap 0 seq_num 45) 2023-10-20T03:35:16.453 INFO:tasks.rados.rados.0.smithi170.stdout:63: read oid 39 snap -1 2023-10-20T03:35:16.454 INFO:tasks.rados.rados.0.smithi170.stdout:63: expect (ObjNum 38 snap 0 seq_num 38) 2023-10-20T03:35:16.454 INFO:tasks.rados.rados.0.smithi170.stdout:64: rmattr oid 23 current snap is 0 2023-10-20T03:35:16.454 INFO:tasks.rados.rados.0.smithi170.stdout:64: done (3 left) 2023-10-20T03:35:16.454 INFO:tasks.rados.rados.0.smithi170.stdout:65: delete oid 47 current snap is 0 2023-10-20T03:35:16.454 INFO:tasks.rados.rados.0.smithi170.stdout:65: done (3 left) 2023-10-20T03:35:16.454 INFO:tasks.rados.rados.0.smithi170.stdout:66: read oid 40 snap -1 2023-10-20T03:35:16.455 INFO:tasks.rados.rados.0.smithi170.stdout:66: expect (ObjNum 39 snap 0 seq_num 39) 2023-10-20T03:35:16.455 INFO:tasks.rados.rados.0.smithi170.stdout:67: rollback oid 4 current snap is 0 2023-10-20T03:35:16.455 INFO:tasks.rados.rados.0.smithi170.stdout:67: done (4 left) 2023-10-20T03:35:16.455 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 37 current snap is 0 2023-10-20T03:35:16.455 INFO:tasks.rados.rados.0.smithi170.stdout:68: seq_num 56 ranges {606208=573440,1179648=376832} 2023-10-20T03:35:16.456 INFO:tasks.rados.rados.0.smithi170.stdout:68: writing smithi17039300-37 from 606208 to 1179648 tid 1 2023-10-20T03:35:16.457 INFO:tasks.rados.rados.0.smithi170.stdout:68: writing smithi17039300-37 from 1179648 to 1556480 tid 2 2023-10-20T03:35:16.457 INFO:tasks.rados.rados.0.smithi170.stdout:69: delete oid 10 current snap is 0 2023-10-20T03:35:16.457 INFO:tasks.rados.rados.0.smithi170.stdout:62: expect (ObjNum 45 snap 0 seq_num 45) 2023-10-20T03:35:16.478 INFO:tasks.rados.rados.0.smithi170.stdout:63: expect (ObjNum 38 snap 0 seq_num 38) 2023-10-20T03:35:16.494 INFO:tasks.rados.rados.0.smithi170.stdout:61: finishing copy_from racing read to smithi17039300-29 2023-10-20T03:35:16.494 INFO:tasks.rados.rados.0.smithi170.stdout:66: expect (ObjNum 39 snap 0 seq_num 39) 2023-10-20T03:35:16.511 INFO:tasks.rados.rados.0.smithi170.stdout:61: finishing copy_from to smithi17039300-29 2023-10-20T03:35:16.512 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 29 v 4 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-10-20T03:35:16.512 INFO:tasks.rados.rados.0.smithi170.stdout:68: finishing write tid 1 to smithi17039300-37 2023-10-20T03:35:16.512 INFO:tasks.rados.rados.0.smithi170.stdout:68: finishing write tid 2 to smithi17039300-37 2023-10-20T03:35:16.512 INFO:tasks.rados.rados.0.smithi170.stdout:68: finishing write tid 3 to smithi17039300-37 2023-10-20T03:35:16.512 INFO:tasks.rados.rados.0.smithi170.stdout:68: finishing write tid 4 to smithi17039300-37 2023-10-20T03:35:16.513 INFO:tasks.rados.rados.0.smithi170.stdout:68: oid 37 updating version 0 to 11 2023-10-20T03:35:16.513 INFO:tasks.rados.rados.0.smithi170.stdout:68: oid 37 version 11 is already newer than 10 2023-10-20T03:35:16.513 INFO:tasks.rados.rados.0.smithi170.stdout:68: oid 37 version 11 is already newer than 9 2023-10-20T03:35:16.513 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 37 v 11 (ObjNum 56 snap 0 seq_num 56) dirty exists 2023-10-20T03:35:16.513 INFO:tasks.rados.rados.0.smithi170.stdout:68: left oid 37 (ObjNum 56 snap 0 seq_num 56) 2023-10-20T03:35:16.514 INFO:tasks.rados.rados.0.smithi170.stdout:61: done (5 left) 2023-10-20T03:35:16.514 INFO:tasks.rados.rados.0.smithi170.stdout:62: done (4 left) 2023-10-20T03:35:16.514 INFO:tasks.rados.rados.0.smithi170.stdout:63: done (3 left) 2023-10-20T03:35:16.514 INFO:tasks.rados.rados.0.smithi170.stdout:66: done (2 left) 2023-10-20T03:35:16.514 INFO:tasks.rados.rados.0.smithi170.stdout:68: done (1 left) 2023-10-20T03:35:16.515 INFO:tasks.rados.rados.0.smithi170.stdout:69: done (0 left) 2023-10-20T03:35:16.515 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 39 current snap is 0 2023-10-20T03:35:16.515 INFO:tasks.rados.rados.0.smithi170.stdout:70: seq_num 58 ranges {638976=606208,1245184=376832} 2023-10-20T03:35:16.518 INFO:tasks.rados.rados.0.smithi170.stdout:70: writing smithi17039300-39 from 638976 to 1245184 tid 1 2023-10-20T03:35:16.520 INFO:tasks.rados.rados.0.smithi170.stdout:70: writing smithi17039300-39 from 1245184 to 1622016 tid 2 2023-10-20T03:35:16.521 INFO:tasks.rados.rados.0.smithi170.stdout:71: snap_create 2023-10-20T03:35:16.541 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:16.537+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:16.543 INFO:tasks.rados.rados.0.smithi170.stdout:70: finishing write tid 1 to smithi17039300-39 2023-10-20T03:35:16.544 INFO:tasks.rados.rados.0.smithi170.stdout:70: finishing write tid 2 to smithi17039300-39 2023-10-20T03:35:16.545 INFO:tasks.rados.rados.0.smithi170.stdout:70: finishing write tid 3 to smithi17039300-39 2023-10-20T03:35:16.549 INFO:tasks.rados.rados.0.smithi170.stdout:70: finishing write tid 4 to smithi17039300-39 2023-10-20T03:35:16.549 INFO:tasks.rados.rados.0.smithi170.stdout:70: oid 39 updating version 0 to 7 2023-10-20T03:35:16.549 INFO:tasks.rados.rados.0.smithi170.stdout:70: oid 39 updating version 7 to 8 2023-10-20T03:35:16.549 INFO:tasks.rados.rados.0.smithi170.stdout:70: oid 39 updating version 8 to 9 2023-10-20T03:35:16.550 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 9 (ObjNum 58 snap 0 seq_num 58) dirty exists 2023-10-20T03:35:16.550 INFO:tasks.rados.rados.0.smithi170.stdout:70: left oid 39 (ObjNum 58 snap 0 seq_num 58) 2023-10-20T03:35:16.566 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:16.567 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:16.580 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-10-20T03:35:16.642 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:16.637+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:16.744 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:16.742+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:16.744 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:16.744 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:16.759 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-10-20T03:35:16.845 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:16.841+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:16.881 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:16.882 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:16.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-10-20T03:35:16.946 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:16.946+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:17.018 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:17.018 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:17.032 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-10-20T03:35:17.046 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:17.046+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:17.148 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:17.146+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:17.151 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:17.152 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:17.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-10-20T03:35:17.182 INFO:teuthology.orchestra.run.smithi064.stdout:get_pg_num_history pg_num_history pg_num_history(e28 pg_nums {1={23=1},2={26=8}} deleted_pools ) 2023-10-20T03:35:17.182 INFO:teuthology.orchestra.run.smithi064.stdout:Importing pgid 1.0 2023-10-20T03:35:17.188 INFO:teuthology.orchestra.run.smithi064.stdout:Write #1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head# 2023-10-20T03:35:17.188 INFO:teuthology.orchestra.run.smithi064.stdout:snapset 0=[]:{} 2023-10-20T03:35:17.190 INFO:teuthology.orchestra.run.smithi064.stdout:Write #1:30306672:devicehealth::main.db.0000000000000000:head# 2023-10-20T03:35:17.190 INFO:teuthology.orchestra.run.smithi064.stdout:snapset 0=[]:{} 2023-10-20T03:35:17.193 INFO:teuthology.orchestra.run.smithi064.stdout:write_pg epoch 26 info 1.0( v 26'157 (0'0,26'157] local-lis/les=23/24 n=2 ec=23/23 lis/c=23/23 les/c/f=24/24/0 sis=23) 2023-10-20T03:35:17.195 INFO:teuthology.orchestra.run.smithi064.stdout:Import successful 2023-10-20T03:35:17.227 INFO:tasks.rados.rados.0.smithi170.stdout:70: done (1 left) 2023-10-20T03:35:17.227 INFO:tasks.rados.rados.0.smithi170.stdout:71: done (0 left) 2023-10-20T03:35:17.227 INFO:tasks.rados.rados.0.smithi170.stdout:72: delete oid 7 current snap is 1 2023-10-20T03:35:17.228 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2023-10-20T03:35:17.231 INFO:tasks.rados.rados.0.smithi170.stdout:72: done (0 left) 2023-10-20T03:35:17.231 INFO:tasks.rados.rados.0.smithi170.stdout:73: read oid 28 snap 0 2023-10-20T03:35:17.231 INFO:tasks.rados.rados.0.smithi170.stdout:73: expect (ObjNum 27 snap 0 seq_num 27) 2023-10-20T03:35:17.231 INFO:tasks.rados.rados.0.smithi170.stdout:74: read oid 2 snap 0 2023-10-20T03:35:17.231 INFO:tasks.rados.rados.0.smithi170.stdout:74: expect (ObjNum 1 snap 0 seq_num 1) 2023-10-20T03:35:17.232 INFO:tasks.rados.rados.0.smithi170.stdout:75: setattr oid 39 current snap is 1 2023-10-20T03:35:17.235 INFO:tasks.rados.rados.0.smithi170.stdout:74: expect (ObjNum 1 snap 0 seq_num 1) 2023-10-20T03:35:17.236 INFO:tasks.rados.rados.0.smithi170.stdout:74: done (2 left) 2023-10-20T03:35:17.236 INFO:tasks.rados.rados.0.smithi170.stdout:76: snap_create 2023-10-20T03:35:17.241 INFO:tasks.rados.rados.0.smithi170.stdout:73: expect (ObjNum 27 snap 0 seq_num 27) 2023-10-20T03:35:17.248 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:17.245+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:17.257 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 11 (ObjNum 58 snap 0 seq_num 58) dirty exists 2023-10-20T03:35:17.293 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:17.294 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:17.307 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-10-20T03:35:17.348 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:17.345+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:17.440 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:17.441 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:17.449 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:17.450+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:17.455 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-10-20T03:35:17.551 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:17.550+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:17.625 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:17.625 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:17.640 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-10-20T03:35:17.652 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:17.649+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:17.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /var/log/ceph/exp.1.0.6 2023-10-20T03:35:17.753 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:17.749+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:17.810 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:17.811 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:17.825 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-10-20T03:35:17.854 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:17.849+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:17.888 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:35:17.946 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:17.946 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:17.955 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:17.953+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:17.962 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-10-20T03:35:18.033 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:18.033 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.053 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:18.053 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.056 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:18.055+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:18.075 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:18.076 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.085 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:18.085 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:18.098 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:18.098 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.100 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-10-20T03:35:18.116 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:18.116 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.138 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:18.139 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.139 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:18.157 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:18.155+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:18.160 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:18.161 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.180 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:18.181 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.202 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:18.203 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.225 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:18.225 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.228 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:18.228 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:18.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:18.229 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [] 2023-10-20T03:35:18.229 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:18.238 INFO:tasks.rados.rados.0.smithi170.stdout:73: done (2 left) 2023-10-20T03:35:18.238 INFO:tasks.rados.rados.0.smithi170.stdout:75: done (1 left) 2023-10-20T03:35:18.238 INFO:tasks.rados.rados.0.smithi170.stdout:76: done (0 left) 2023-10-20T03:35:18.239 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 42 current snap is 2 2023-10-20T03:35:18.239 INFO:tasks.rados.rados.0.smithi170.stdout:77: seq_num 60 ranges {704512=638976,1343488=376832} 2023-10-20T03:35:18.244 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-10-20T03:35:18.251 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:18.252 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.260 INFO:tasks.rados.rados.0.smithi170.stdout:77: writing smithi17039300-42 from 704512 to 1343488 tid 1 2023-10-20T03:35:18.267 INFO:tasks.rados.rados.0.smithi170.stdout:77: writing smithi17039300-42 from 1343488 to 1720320 tid 2 2023-10-20T03:35:18.267 INFO:tasks.rados.rados.0.smithi170.stdout:78: rollback oid 30 current snap is 2 2023-10-20T03:35:18.267 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 30 to 0 2023-10-20T03:35:18.267 INFO:tasks.rados.rados.0.smithi170.stdout:79: delete oid 27 current snap is 2 2023-10-20T03:35:18.274 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:18.275 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.275 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:18.274+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:18.279 INFO:tasks.rados.rados.0.smithi170.stdout:79: done (2 left) 2023-10-20T03:35:18.280 INFO:tasks.rados.rados.0.smithi170.stdout:80: delete oid 43 current snap is 2 2023-10-20T03:35:18.283 INFO:tasks.rados.rados.0.smithi170.stdout:78: finishing rollback tid 0 to smithi17039300-30 2023-10-20T03:35:18.284 INFO:tasks.rados.rados.0.smithi170.stdout:78: finishing rollback tid 1 to smithi17039300-30 2023-10-20T03:35:18.288 INFO:tasks.rados.rados.0.smithi170.stdout:80: done (2 left) 2023-10-20T03:35:18.288 INFO:tasks.rados.rados.0.smithi170.stdout:81: snap_create 2023-10-20T03:35:18.290 INFO:tasks.rados.rados.0.smithi170.stdout:78: finishing rollback tid 2 to smithi17039300-30 2023-10-20T03:35:18.291 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 18 (ObjNum 29 snap 0 seq_num 29) dirty exists 2023-10-20T03:35:18.293 INFO:tasks.rados.rados.0.smithi170.stdout:77: finishing write tid 1 to smithi17039300-42 2023-10-20T03:35:18.295 INFO:tasks.rados.rados.0.smithi170.stdout:77: finishing write tid 2 to smithi17039300-42 2023-10-20T03:35:18.296 INFO:tasks.rados.rados.0.smithi170.stdout:77: finishing write tid 3 to smithi17039300-42 2023-10-20T03:35:18.300 INFO:tasks.rados.rados.0.smithi170.stdout:77: finishing write tid 4 to smithi17039300-42 2023-10-20T03:35:18.300 INFO:tasks.rados.rados.0.smithi170.stdout:77: oid 42 updating version 0 to 15 2023-10-20T03:35:18.301 INFO:tasks.rados.rados.0.smithi170.stdout:77: oid 42 version 15 is already newer than 14 2023-10-20T03:35:18.301 INFO:tasks.rados.rados.0.smithi170.stdout:77: oid 42 version 15 is already newer than 13 2023-10-20T03:35:18.301 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 42 v 15 (ObjNum 60 snap 2 seq_num 60) dirty exists 2023-10-20T03:35:18.301 INFO:tasks.rados.rados.0.smithi170.stdout:77: left oid 42 (ObjNum 60 snap 2 seq_num 60) 2023-10-20T03:35:18.302 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:18.302 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.322 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:18.322 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.342 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:18.343 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:35:18.347 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:18.347 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:18.358 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:18.357+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:18.390 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:18.390 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:18.390 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:18.391 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:18.404 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-10-20T03:35:18.459 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:18.459+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:18.561 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:18.559+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:18.565 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:18.565 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:18.565 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:35:18.565 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:35:18.566 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:18.580 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-10-20T03:35:18.661 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:18.657+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:18.702 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:18.702 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:18.703 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:18.703 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:35:18.703 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.703 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=21, reqid=client.4570.0:198, at_version=36'13, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:18.704 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:18.704 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.435936343, 2023-10-20T03:35:18.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086318609999999994, 2023-10-20T03:35:18.704 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:18.705 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:18.705 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:18.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.705 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:18.705 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:18.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:18.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:18.707 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:18.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:18.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.707 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:18.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:18.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.58e-07 2023-10-20T03:35:18.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.709 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:18.709 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.267902+0000", 2023-10-20T03:35:18.709 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014514409999999999 2023-10-20T03:35:18.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.709 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.710 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:18.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.267916+0000", 2023-10-20T03:35:18.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4664999999999999e-05 2023-10-20T03:35:18.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.711 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:18.711 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.267931+0000", 2023-10-20T03:35:18.711 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4192999999999999e-05 2023-10-20T03:35:18.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.712 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.712 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:18.712 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.268007+0000", 2023-10-20T03:35:18.712 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6058999999999998e-05 2023-10-20T03:35:18.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.713 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:18.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.268017+0000", 2023-10-20T03:35:18.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0648999999999999e-05 2023-10-20T03:35:18.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:18.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275053+0000", 2023-10-20T03:35:18.715 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0070355699999999997 2023-10-20T03:35:18.715 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.715 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:18.715 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275082+0000", 2023-10-20T03:35:18.716 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8926000000000001e-05 2023-10-20T03:35:18.716 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:18.716 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:18.716 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:18.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.717 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=22, reqid=client.4570.0:199, at_version=36'14, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:18.717 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.284006+0000", 2023-10-20T03:35:18.717 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.41837989599999997, 2023-10-20T03:35:18.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0061959429999999998, 2023-10-20T03:35:18.718 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:18.718 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:18.718 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:18.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:18.719 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284006+0000", 2023-10-20T03:35:18.719 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:18.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:18.720 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284006+0000", 2023-10-20T03:35:18.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:18.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.721 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:18.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284008+0000", 2023-10-20T03:35:18.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.398e-06 2023-10-20T03:35:18.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.722 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.722 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:18.722 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284090+0000", 2023-10-20T03:35:18.722 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2686000000000001e-05 2023-10-20T03:35:18.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.723 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.723 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:18.723 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284111+0000", 2023-10-20T03:35:18.723 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0363000000000001e-05 2023-10-20T03:35:18.724 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.724 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.724 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:18.724 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284123+0000", 2023-10-20T03:35:18.725 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2213e-05 2023-10-20T03:35:18.725 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.725 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.725 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:18.725 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284222+0000", 2023-10-20T03:35:18.726 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8587999999999999e-05 2023-10-20T03:35:18.726 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.726 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.726 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:18.726 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284237+0000", 2023-10-20T03:35:18.727 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.541e-05 2023-10-20T03:35:18.727 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.727 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.727 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:18.728 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.290159+0000", 2023-10-20T03:35:18.728 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0059216299999999998 2023-10-20T03:35:18.728 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.728 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.728 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:18.729 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.290202+0000", 2023-10-20T03:35:18.729 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3655000000000001e-05 2023-10-20T03:35:18.729 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:18.729 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:18.729 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:18.730 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.730 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.730 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=23, reqid=client.4570.0:200, at_version=36'15, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:18.730 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.293836+0000", 2023-10-20T03:35:18.730 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.40855076499999998, 2023-10-20T03:35:18.731 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00083768799999999995, 2023-10-20T03:35:18.731 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:18.731 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:18.731 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:18.731 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.732 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:18.732 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293836+0000", 2023-10-20T03:35:18.732 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:18.732 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.733 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.733 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:18.733 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293836+0000", 2023-10-20T03:35:18.733 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:18.734 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.734 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.734 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:18.734 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293836+0000", 2023-10-20T03:35:18.735 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1399999999999995e-07 2023-10-20T03:35:18.735 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.735 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.735 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:18.735 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293862+0000", 2023-10-20T03:35:18.736 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5675999999999999e-05 2023-10-20T03:35:18.736 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.736 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.736 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:18.736 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293876+0000", 2023-10-20T03:35:18.737 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3774000000000001e-05 2023-10-20T03:35:18.737 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.737 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.737 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:18.737 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293885+0000", 2023-10-20T03:35:18.738 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3309999999999993e-06 2023-10-20T03:35:18.738 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.738 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.738 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:18.739 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293974+0000", 2023-10-20T03:35:18.739 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8782999999999993e-05 2023-10-20T03:35:18.739 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.739 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.739 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:18.740 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293989+0000", 2023-10-20T03:35:18.740 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4559000000000001e-05 2023-10-20T03:35:18.740 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.740 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.740 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:18.741 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.294644+0000", 2023-10-20T03:35:18.741 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00065499600000000003 2023-10-20T03:35:18.741 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.741 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.741 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:18.742 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.294673+0000", 2023-10-20T03:35:18.742 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9655e-05 2023-10-20T03:35:18.742 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:18.742 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:18.743 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:18.743 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.743 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.743 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=24, reqid=client.4570.0:200, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-10-20T03:35:18.743 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.297883+0000", 2023-10-20T03:35:18.744 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.40450371400000001, 2023-10-20T03:35:18.744 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000851373, 2023-10-20T03:35:18.744 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:18.744 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:18.745 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:18.745 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.745 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:18.745 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297883+0000", 2023-10-20T03:35:18.746 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:18.746 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.746 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.746 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:18.747 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297883+0000", 2023-10-20T03:35:18.747 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:18.747 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.747 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.747 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:18.748 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297883+0000", 2023-10-20T03:35:18.748 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1999999999999999e-07 2023-10-20T03:35:18.748 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.748 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.749 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:18.749 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297895+0000", 2023-10-20T03:35:18.749 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1677e-05 2023-10-20T03:35:18.749 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.749 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.750 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:18.750 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297913+0000", 2023-10-20T03:35:18.750 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8420999999999999e-05 2023-10-20T03:35:18.750 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.751 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.751 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:18.751 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297923+0000", 2023-10-20T03:35:18.751 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0054999999999999e-05 2023-10-20T03:35:18.751 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.752 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.752 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:18.752 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297986+0000", 2023-10-20T03:35:18.752 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2674000000000003e-05 2023-10-20T03:35:18.753 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.753 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.753 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:18.753 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297995+0000", 2023-10-20T03:35:18.753 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8850000000000005e-06 2023-10-20T03:35:18.754 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.754 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.754 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:18.754 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.298695+0000", 2023-10-20T03:35:18.755 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00070026500000000005 2023-10-20T03:35:18.755 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:18.755 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:18.755 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:18.755 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.298734+0000", 2023-10-20T03:35:18.756 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8676e-05 2023-10-20T03:35:18.756 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:18.756 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:18.756 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:18.756 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:18.757 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:18.757 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:18.758 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-10-20T03:35:18.762 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:18.759+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:18.863 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:18.863+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:18.886 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:18.886 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:18.886 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:18.886 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:18.900 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-10-20T03:35:18.964 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:18.963+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:19.026 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:19.027 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:19.027 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:35:19.027 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:35:19.027 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:19.041 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-10-20T03:35:19.065 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:19.061+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:19.160 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:19.161 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:19.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:19.161 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:35:19.162 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.162 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 36/31 ECSubWrite(tid=59, reqid=client.4570.0:202, at_version=36'16, trim_to=0'0, roll_forward_to=34'14))", 2023-10-20T03:35:19.162 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.276032+0000", 2023-10-20T03:35:19.162 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.884179567, 2023-10-20T03:35:19.163 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001883289, 2023-10-20T03:35:19.163 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:19.163 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:19.163 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:19.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.164 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:19.164 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.276032+0000", 2023-10-20T03:35:19.164 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:19.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.165 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:19.165 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.276032+0000", 2023-10-20T03:35:19.165 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:19.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:19.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.276033+0000", 2023-10-20T03:35:19.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5799999999999999e-07 2023-10-20T03:35:19.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:19.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.276058+0000", 2023-10-20T03:35:19.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5704999999999999e-05 2023-10-20T03:35:19.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:19.169 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.276071+0000", 2023-10-20T03:35:19.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2948999999999999e-05 2023-10-20T03:35:19.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.170 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:19.170 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.276080+0000", 2023-10-20T03:35:19.170 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8640000000000002e-06 2023-10-20T03:35:19.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.171 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:19.171 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.276148+0000", 2023-10-20T03:35:19.171 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7835000000000004e-05 2023-10-20T03:35:19.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:19.172 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.276155+0000", 2023-10-20T03:35:19.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.737e-06 2023-10-20T03:35:19.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.173 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:19.173 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.277870+0000", 2023-10-20T03:35:19.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0017149470000000001 2023-10-20T03:35:19.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:19.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.277915+0000", 2023-10-20T03:35:19.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5694e-05 2023-10-20T03:35:19.175 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:19.175 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:19.175 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:19.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.176 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.176 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 36/31 ECSubWrite(tid=60, reqid=client.4570.0:203, at_version=36'17, trim_to=0'0, roll_forward_to=34'14))", 2023-10-20T03:35:19.176 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.278276+0000", 2023-10-20T03:35:19.176 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.88193561899999995, 2023-10-20T03:35:19.176 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019051459999999999, 2023-10-20T03:35:19.177 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:19.177 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:19.177 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:19.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.177 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:19.177 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278276+0000", 2023-10-20T03:35:19.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:19.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.178 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:19.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278276+0000", 2023-10-20T03:35:19.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:19.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:19.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278277+0000", 2023-10-20T03:35:19.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0500000000000003e-07 2023-10-20T03:35:19.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:19.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278297+0000", 2023-10-20T03:35:19.181 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0404e-05 2023-10-20T03:35:19.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.182 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:19.182 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278311+0000", 2023-10-20T03:35:19.183 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3407999999999999e-05 2023-10-20T03:35:19.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.183 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:19.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278319+0000", 2023-10-20T03:35:19.184 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1380000000000007e-06 2023-10-20T03:35:19.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.184 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:19.185 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278378+0000", 2023-10-20T03:35:19.185 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9001e-05 2023-10-20T03:35:19.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.185 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:19.185 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278389+0000", 2023-10-20T03:35:19.186 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.119e-05 2023-10-20T03:35:19.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.186 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.186 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:19.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.280140+0000", 2023-10-20T03:35:19.187 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0017506620000000001 2023-10-20T03:35:19.187 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.187 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:19.188 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.280181+0000", 2023-10-20T03:35:19.188 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1737999999999999e-05 2023-10-20T03:35:19.188 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:19.188 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:19.189 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:19.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.189 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.189 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 36/31 ECSubWrite(tid=61, reqid=client.4570.0:204, at_version=36'18, trim_to=0'0, roll_forward_to=34'14))", 2023-10-20T03:35:19.189 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.279787+0000", 2023-10-20T03:35:19.190 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.88042435600000002, 2023-10-20T03:35:19.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0026782350000000002, 2023-10-20T03:35:19.190 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:19.190 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:19.190 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:19.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:19.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279787+0000", 2023-10-20T03:35:19.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:19.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.192 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:19.192 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279787+0000", 2023-10-20T03:35:19.192 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:19.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.192 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.193 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:19.193 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279788+0000", 2023-10-20T03:35:19.193 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0200000000000001e-07 2023-10-20T03:35:19.193 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.193 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.194 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:19.194 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279812+0000", 2023-10-20T03:35:19.194 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3714999999999999e-05 2023-10-20T03:35:19.194 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.194 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.195 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:19.195 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279826+0000", 2023-10-20T03:35:19.195 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4531e-05 2023-10-20T03:35:19.195 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.195 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.196 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:19.196 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279835+0000", 2023-10-20T03:35:19.196 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2840000000000006e-06 2023-10-20T03:35:19.196 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.196 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.197 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:19.197 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279900+0000", 2023-10-20T03:35:19.197 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5265000000000006e-05 2023-10-20T03:35:19.197 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.197 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.198 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:19.198 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279910+0000", 2023-10-20T03:35:19.198 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0159e-05 2023-10-20T03:35:19.198 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.198 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.199 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:19.199 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.282417+0000", 2023-10-20T03:35:19.199 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025064779999999999 2023-10-20T03:35:19.199 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.199 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.200 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:19.200 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.282466+0000", 2023-10-20T03:35:19.200 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9101000000000003e-05 2023-10-20T03:35:19.200 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:19.200 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:19.201 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:19.201 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.201 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.201 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 36/31 ECSubWrite(tid=62, reqid=client.4570.0:204, at_version=0'0, trim_to=0'0, roll_forward_to=36'18))", 2023-10-20T03:35:19.201 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.292197+0000", 2023-10-20T03:35:19.202 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.86801500399999998, 2023-10-20T03:35:19.202 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001048979, 2023-10-20T03:35:19.202 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:19.202 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:19.202 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:19.202 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.203 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:19.203 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292197+0000", 2023-10-20T03:35:19.203 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:19.203 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.203 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.204 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:19.204 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292197+0000", 2023-10-20T03:35:19.204 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:19.204 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.205 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.205 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:19.205 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292199+0000", 2023-10-20T03:35:19.205 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9290000000000001e-06 2023-10-20T03:35:19.205 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.205 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.206 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:19.206 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292223+0000", 2023-10-20T03:35:19.206 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4649000000000001e-05 2023-10-20T03:35:19.206 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.206 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.207 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:19.207 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292252+0000", 2023-10-20T03:35:19.207 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.828e-05 2023-10-20T03:35:19.207 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.207 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.208 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:19.208 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292267+0000", 2023-10-20T03:35:19.208 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5316999999999999e-05 2023-10-20T03:35:19.208 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.208 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.208 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:19.209 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292351+0000", 2023-10-20T03:35:19.209 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4528000000000001e-05 2023-10-20T03:35:19.209 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.209 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.209 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:19.210 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292363+0000", 2023-10-20T03:35:19.210 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1979e-05 2023-10-20T03:35:19.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.210 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.210 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:19.211 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293218+0000", 2023-10-20T03:35:19.211 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00085484500000000002 2023-10-20T03:35:19.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:19.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:19.211 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:19.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293246+0000", 2023-10-20T03:35:19.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7452000000000001e-05 2023-10-20T03:35:19.212 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:19.212 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:19.212 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:19.212 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:19.213 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:19.213 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:19.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-10-20T03:35:19.215 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:19.213+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:19.255 INFO:tasks.rados.rados.0.smithi170.stdout:77: done (2 left) 2023-10-20T03:35:19.255 INFO:tasks.rados.rados.0.smithi170.stdout:78: done (1 left) 2023-10-20T03:35:19.256 INFO:tasks.rados.rados.0.smithi170.stdout:81: done (0 left) 2023-10-20T03:35:19.256 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 5 current snap is 3 2023-10-20T03:35:19.256 INFO:tasks.rados.rados.0.smithi170.stdout:82: seq_num 63 ranges {81920=688128,770048=376832} 2023-10-20T03:35:19.258 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-10-20T03:35:19.267 INFO:tasks.rados.rados.0.smithi170.stdout:82: writing smithi17039300-5 from 81920 to 770048 tid 1 2023-10-20T03:35:19.275 INFO:tasks.rados.rados.0.smithi170.stdout:82: writing smithi17039300-5 from 770048 to 1146880 tid 2 2023-10-20T03:35:19.276 INFO:tasks.rados.rados.0.smithi170.stdout:83: read oid 29 snap 0 2023-10-20T03:35:19.276 INFO:tasks.rados.rados.0.smithi170.stdout:83: expect (ObjNum 25 snap 0 seq_num 25) 2023-10-20T03:35:19.276 INFO:tasks.rados.rados.0.smithi170.stdout:84: snap_remove snap 2 2023-10-20T03:35:19.315 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:19.313+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:19.356 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:19.356 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:19.357 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:35:19.357 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:19.370 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-10-20T03:35:19.415 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:19.414+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:19.517 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:19.516+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:19.540 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:19.541 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:19.541 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:35:19.541 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:35:19.542 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:19.554 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-10-20T03:35:19.617 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:19.616+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:19.677 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:19.677 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:35:19.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:35:19.678 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2023-10-20T03:35:19.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.678 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.fs6 36/31 ECSubWrite(tid=32, reqid=client.4570.0:206, at_version=36'13, trim_to=0'0, roll_forward_to=34'11))", 2023-10-20T03:35:19.679 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:18.281398+0000", 2023-10-20T03:35:19.679 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.3960066929999999, 2023-10-20T03:35:19.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035446459999999998, 2023-10-20T03:35:19.680 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:19.680 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:19.680 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:19.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:19.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.281398+0000", 2023-10-20T03:35:19.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:19.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:19.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.281398+0000", 2023-10-20T03:35:19.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:19.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:19.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.281399+0000", 2023-10-20T03:35:19.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2999999999999999e-07 2023-10-20T03:35:19.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:19.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.281498+0000", 2023-10-20T03:35:19.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9112999999999998e-05 2023-10-20T03:35:19.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:19.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.281533+0000", 2023-10-20T03:35:19.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4952e-05 2023-10-20T03:35:19.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:19.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.281570+0000", 2023-10-20T03:35:19.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7036999999999998e-05 2023-10-20T03:35:19.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:19.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.281742+0000", 2023-10-20T03:35:19.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017192600000000001 2023-10-20T03:35:19.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:19.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.281762+0000", 2023-10-20T03:35:19.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9899e-05 2023-10-20T03:35:19.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:19.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.284867+0000", 2023-10-20T03:35:19.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0031052670000000001 2023-10-20T03:35:19.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:19.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.284943+0000", 2023-10-20T03:35:19.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5622000000000002e-05 2023-10-20T03:35:19.692 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:19.692 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:19.692 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.fs6 36/31 ECSubWrite(tid=33, reqid=client.4570.0:206, at_version=0'0, trim_to=0'0, roll_forward_to=36'13))", 2023-10-20T03:35:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:18.288794+0000", 2023-10-20T03:35:19.693 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.388610948, 2023-10-20T03:35:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001844589, 2023-10-20T03:35:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:19.694 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:19.695 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:19.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:19.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.288794+0000", 2023-10-20T03:35:19.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:19.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:19.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.288794+0000", 2023-10-20T03:35:19.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:19.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:19.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.288796+0000", 2023-10-20T03:35:19.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.626e-06 2023-10-20T03:35:19.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:19.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.288817+0000", 2023-10-20T03:35:19.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0893000000000001e-05 2023-10-20T03:35:19.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:19.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.288847+0000", 2023-10-20T03:35:19.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0433000000000002e-05 2023-10-20T03:35:19.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:19.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.288862+0000", 2023-10-20T03:35:19.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4993e-05 2023-10-20T03:35:19.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:19.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.289021+0000", 2023-10-20T03:35:19.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015938599999999999 2023-10-20T03:35:19.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:19.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.289051+0000", 2023-10-20T03:35:19.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9873e-05 2023-10-20T03:35:19.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:19.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.290568+0000", 2023-10-20T03:35:19.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0015163589999999999 2023-10-20T03:35:19.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:19.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.290639+0000", 2023-10-20T03:35:19.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1025999999999994e-05 2023-10-20T03:35:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:19.709 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:19.710 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:19.711 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-10-20T03:35:19.718 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:19.717+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:19.819 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:19.812+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:19.839 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:19.839 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:19.840 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:35:19.840 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:19.855 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-10-20T03:35:19.920 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:19.917+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:19.979 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:19.979 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:19.979 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:35:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:35:19.980 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:19.993 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-10-20T03:35:20.021 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:20.017+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:20.122 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:20.117+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:20.126 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:20.127 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:35:20.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:35:20.127 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2023-10-20T03:35:20.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.127 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 36/31 ECSubWrite(tid=21, reqid=client.4570.0:205, at_version=36'12, trim_to=0'0, roll_forward_to=34'9))", 2023-10-20T03:35:20.128 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:18.270223+0000", 2023-10-20T03:35:20.128 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.855750338, 2023-10-20T03:35:20.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037386780000000001, 2023-10-20T03:35:20.128 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:20.128 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:20.129 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:20.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:20.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.270223+0000", 2023-10-20T03:35:20.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.130 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:20.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.270223+0000", 2023-10-20T03:35:20.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:20.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.270223+0000", 2023-10-20T03:35:20.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3000000000000002e-07 2023-10-20T03:35:20.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:20.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.270297+0000", 2023-10-20T03:35:20.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3819000000000006e-05 2023-10-20T03:35:20.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:20.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.270331+0000", 2023-10-20T03:35:20.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3859999999999998e-05 2023-10-20T03:35:20.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:20.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.270349+0000", 2023-10-20T03:35:20.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7736000000000001e-05 2023-10-20T03:35:20.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.134 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:20.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.270533+0000", 2023-10-20T03:35:20.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018463900000000001 2023-10-20T03:35:20.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.135 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:20.135 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.270566+0000", 2023-10-20T03:35:20.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2060000000000001e-05 2023-10-20T03:35:20.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:20.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.273862+0000", 2023-10-20T03:35:20.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0032963720000000001 2023-10-20T03:35:20.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:20.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.273962+0000", 2023-10-20T03:35:20.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9862e-05 2023-10-20T03:35:20.137 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.137 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:20.137 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.138 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 36/31 ECSubWrite(tid=22, reqid=client.4570.0:205, at_version=0'0, trim_to=0'0, roll_forward_to=36'12))", 2023-10-20T03:35:20.138 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:18.280898+0000", 2023-10-20T03:35:20.139 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.845075378, 2023-10-20T03:35:20.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0024799470000000001, 2023-10-20T03:35:20.139 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:20.139 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:20.139 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:20.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:20.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.280898+0000", 2023-10-20T03:35:20.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:20.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.280898+0000", 2023-10-20T03:35:20.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:20.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.280900+0000", 2023-10-20T03:35:20.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6530000000000001e-06 2023-10-20T03:35:20.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:20.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.280929+0000", 2023-10-20T03:35:20.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8972999999999999e-05 2023-10-20T03:35:20.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:20.143 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.280958+0000", 2023-10-20T03:35:20.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9669999999999999e-05 2023-10-20T03:35:20.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:20.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.280974+0000", 2023-10-20T03:35:20.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5452999999999999e-05 2023-10-20T03:35:20.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:20.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.281144+0000", 2023-10-20T03:35:20.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017053200000000001 2023-10-20T03:35:20.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:20.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.281175+0000", 2023-10-20T03:35:20.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0942999999999998e-05 2023-10-20T03:35:20.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:20.147 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.283277+0000", 2023-10-20T03:35:20.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021019609999999998 2023-10-20T03:35:20.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.147 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:20.147 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:18.283378+0000", 2023-10-20T03:35:20.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100762 2023-10-20T03:35:20.148 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.148 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:20.148 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.149 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=23, reqid=client.4570.0:208, at_version=37'14, trim_to=0'0, roll_forward_to=36'12))", 2023-10-20T03:35:20.149 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:19.272177+0000", 2023-10-20T03:35:20.149 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.85379636800000003, 2023-10-20T03:35:20.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010004097, 2023-10-20T03:35:20.149 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:20.149 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:20.150 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:20.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:20.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272177+0000", 2023-10-20T03:35:20.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:20.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272177+0000", 2023-10-20T03:35:20.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:20.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272177+0000", 2023-10-20T03:35:20.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5800000000000001e-07 2023-10-20T03:35:20.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:20.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272691+0000", 2023-10-20T03:35:20.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051351599999999999 2023-10-20T03:35:20.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:20.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272709+0000", 2023-10-20T03:35:20.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8108999999999999e-05 2023-10-20T03:35:20.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.154 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:20.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272738+0000", 2023-10-20T03:35:20.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9246000000000001e-05 2023-10-20T03:35:20.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:20.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272855+0000", 2023-10-20T03:35:20.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011678 2023-10-20T03:35:20.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:20.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272869+0000", 2023-10-20T03:35:20.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4409999999999999e-05 2023-10-20T03:35:20.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:20.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282065+0000", 2023-10-20T03:35:20.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0091960219999999999 2023-10-20T03:35:20.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:20.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282181+0000", 2023-10-20T03:35:20.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011575599999999999 2023-10-20T03:35:20.158 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.158 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:20.158 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.159 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpRead(3.5s2 37/31 ECSubRead(tid=11, to_read={3:aed1dba0:::smithi17039300-29:head=0,106496,0}, subchunks={3:aed1dba0:::smithi17039300-29:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:20.159 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:19.282027+0000", 2023-10-20T03:35:20.159 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.84394663599999997, 2023-10-20T03:35:20.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072711800000000001, 2023-10-20T03:35:20.160 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:20.160 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:20.160 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:20.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:20.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282027+0000", 2023-10-20T03:35:20.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:20.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282027+0000", 2023-10-20T03:35:20.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:20.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282027+0000", 2023-10-20T03:35:20.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7099999999999998e-07 2023-10-20T03:35:20.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:20.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282039+0000", 2023-10-20T03:35:20.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1877e-05 2023-10-20T03:35:20.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:20.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282059+0000", 2023-10-20T03:35:20.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9848999999999999e-05 2023-10-20T03:35:20.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:20.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282075+0000", 2023-10-20T03:35:20.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6433999999999999e-05 2023-10-20T03:35:20.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:20.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282185+0000", 2023-10-20T03:35:20.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000110115 2023-10-20T03:35:20.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:20.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282754+0000", 2023-10-20T03:35:20.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00056857200000000004 2023-10-20T03:35:20.167 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.167 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:20.167 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.168 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpRead(3.5s2 37/31 ECSubRead(tid=12, to_read={3:aed1dba0:::smithi17039300-29:head=0,106496,0}, subchunks={3:aed1dba0:::smithi17039300-29:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:20.168 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:19.282749+0000", 2023-10-20T03:35:20.168 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.84322489499999997, 2023-10-20T03:35:20.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025996899999999998, 2023-10-20T03:35:20.169 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:20.169 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:20.169 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:20.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:20.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282749+0000", 2023-10-20T03:35:20.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.170 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:20.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282749+0000", 2023-10-20T03:35:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282749+0000", 2023-10-20T03:35:20.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6300000000000004e-07 2023-10-20T03:35:20.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:20.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282761+0000", 2023-10-20T03:35:20.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.186e-05 2023-10-20T03:35:20.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:20.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282776+0000", 2023-10-20T03:35:20.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4637999999999999e-05 2023-10-20T03:35:20.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.174 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:20.174 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282784+0000", 2023-10-20T03:35:20.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5359999999999992e-06 2023-10-20T03:35:20.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.174 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:20.175 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282862+0000", 2023-10-20T03:35:20.175 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8059999999999995e-05 2023-10-20T03:35:20.175 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.175 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.175 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:20.175 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.283008+0000", 2023-10-20T03:35:20.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000146012 2023-10-20T03:35:20.176 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.176 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:20.176 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.177 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpRead(3.5s2 37/31 ECSubRead(tid=13, to_read={3:aed1dba0:::smithi17039300-29:head=0,106496,0}, subchunks={3:aed1dba0:::smithi17039300-29:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:20.177 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:19.283355+0000", 2023-10-20T03:35:20.177 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.84261811200000003, 2023-10-20T03:35:20.177 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024179, 2023-10-20T03:35:20.177 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:20.178 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:20.178 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:20.178 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.178 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:20.178 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.283355+0000", 2023-10-20T03:35:20.178 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.179 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:20.179 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.283355+0000", 2023-10-20T03:35:20.179 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:20.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.283356+0000", 2023-10-20T03:35:20.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.23e-07 2023-10-20T03:35:20.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.181 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:20.181 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.283367+0000", 2023-10-20T03:35:20.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0842000000000001e-05 2023-10-20T03:35:20.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:20.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.283383+0000", 2023-10-20T03:35:20.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6361000000000001e-05 2023-10-20T03:35:20.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:20.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.283392+0000", 2023-10-20T03:35:20.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.456e-06 2023-10-20T03:35:20.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:20.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.283484+0000", 2023-10-20T03:35:20.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1797000000000002e-05 2023-10-20T03:35:20.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:20.184 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.283597+0000", 2023-10-20T03:35:20.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000113511 2023-10-20T03:35:20.185 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.185 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:20.185 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.185 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=24, reqid=client.4570.0:209, at_version=37'15, trim_to=0'0, roll_forward_to=36'12))", 2023-10-20T03:35:20.186 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:19.288930+0000", 2023-10-20T03:35:20.186 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.837043541, 2023-10-20T03:35:20.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049673369999999996, 2023-10-20T03:35:20.186 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:20.186 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:20.187 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:20.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:20.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.288930+0000", 2023-10-20T03:35:20.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:20.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.288930+0000", 2023-10-20T03:35:20.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:20.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.288932+0000", 2023-10-20T03:35:20.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1789999999999998e-06 2023-10-20T03:35:20.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:20.190 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.289148+0000", 2023-10-20T03:35:20.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000215465 2023-10-20T03:35:20.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.190 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:20.190 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.289179+0000", 2023-10-20T03:35:20.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1744e-05 2023-10-20T03:35:20.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:20.191 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.289203+0000", 2023-10-20T03:35:20.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3932999999999999e-05 2023-10-20T03:35:20.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:20.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.289297+0000", 2023-10-20T03:35:20.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4096000000000002e-05 2023-10-20T03:35:20.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:20.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.289307+0000", 2023-10-20T03:35:20.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5040000000000008e-06 2023-10-20T03:35:20.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.194 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:20.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.293842+0000", 2023-10-20T03:35:20.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045355079999999997 2023-10-20T03:35:20.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.195 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:20.195 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.293897+0000", 2023-10-20T03:35:20.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4907999999999997e-05 2023-10-20T03:35:20.195 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.195 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:20.195 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.196 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.196 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=25, reqid=client.4570.0:209, at_version=0'0, trim_to=0'0, roll_forward_to=37'15))", 2023-10-20T03:35:20.196 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:19.299894+0000", 2023-10-20T03:35:20.196 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.82607976000000005, 2023-10-20T03:35:20.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001343543, 2023-10-20T03:35:20.197 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:20.197 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:20.197 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:20.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:20.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.299894+0000", 2023-10-20T03:35:20.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.198 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:20.198 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.299894+0000", 2023-10-20T03:35:20.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:20.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.299895+0000", 2023-10-20T03:35:20.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.347e-06 2023-10-20T03:35:20.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:20.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.299916+0000", 2023-10-20T03:35:20.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1472999999999999e-05 2023-10-20T03:35:20.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.201 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:20.201 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.299942+0000", 2023-10-20T03:35:20.201 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5610999999999999e-05 2023-10-20T03:35:20.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.201 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:20.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.299954+0000", 2023-10-20T03:35:20.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2394e-05 2023-10-20T03:35:20.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:20.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.300043+0000", 2023-10-20T03:35:20.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8659000000000006e-05 2023-10-20T03:35:20.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.203 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:20.203 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.300059+0000", 2023-10-20T03:35:20.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6050999999999999e-05 2023-10-20T03:35:20.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.204 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:20.204 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.301194+0000", 2023-10-20T03:35:20.204 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0011348020000000001 2023-10-20T03:35:20.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:20.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.301237+0000", 2023-10-20T03:35:20.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3205999999999999e-05 2023-10-20T03:35:20.205 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.206 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:20.206 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.206 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=26, reqid=client.4570.0:210, at_version=37'16, trim_to=0'0, roll_forward_to=37'15))", 2023-10-20T03:35:20.206 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:19.301185+0000", 2023-10-20T03:35:20.207 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.82478827700000001, 2023-10-20T03:35:20.207 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0030485730000000002, 2023-10-20T03:35:20.207 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:20.207 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:20.207 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:20.207 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.208 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:20.208 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.301185+0000", 2023-10-20T03:35:20.208 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.208 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:20.209 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.301185+0000", 2023-10-20T03:35:20.209 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.209 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:20.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.301187+0000", 2023-10-20T03:35:20.210 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7770000000000001e-06 2023-10-20T03:35:20.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.210 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:20.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.301242+0000", 2023-10-20T03:35:20.211 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4789e-05 2023-10-20T03:35:20.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.211 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:20.211 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.301273+0000", 2023-10-20T03:35:20.211 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1272999999999997e-05 2023-10-20T03:35:20.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.212 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:20.212 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.301304+0000", 2023-10-20T03:35:20.212 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1384000000000003e-05 2023-10-20T03:35:20.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.213 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:20.213 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.301390+0000", 2023-10-20T03:35:20.213 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5161000000000007e-05 2023-10-20T03:35:20.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:20.214 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.301398+0000", 2023-10-20T03:35:20.214 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9770000000000006e-06 2023-10-20T03:35:20.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:20.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.304195+0000", 2023-10-20T03:35:20.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0027966279999999998 2023-10-20T03:35:20.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:20.216 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.304234+0000", 2023-10-20T03:35:20.216 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8584000000000002e-05 2023-10-20T03:35:20.216 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.216 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:20.216 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.217 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=27, reqid=client.4570.0:210, at_version=0'0, trim_to=0'0, roll_forward_to=37'16))", 2023-10-20T03:35:20.217 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:19.305365+0000", 2023-10-20T03:35:20.217 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.82060856400000004, 2023-10-20T03:35:20.217 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014814089999999999, 2023-10-20T03:35:20.218 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:20.218 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:20.218 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:20.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.218 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:20.218 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.305365+0000", 2023-10-20T03:35:20.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.219 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:20.219 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.305365+0000", 2023-10-20T03:35:20.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.220 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:20.220 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.305366+0000", 2023-10-20T03:35:20.220 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9700000000000005e-07 2023-10-20T03:35:20.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.221 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:20.221 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.305385+0000", 2023-10-20T03:35:20.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9040000000000001e-05 2023-10-20T03:35:20.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.222 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:20.222 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.305405+0000", 2023-10-20T03:35:20.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0285e-05 2023-10-20T03:35:20.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.222 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:20.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.305420+0000", 2023-10-20T03:35:20.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.501e-05 2023-10-20T03:35:20.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:20.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.305493+0000", 2023-10-20T03:35:20.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3176000000000005e-05 2023-10-20T03:35:20.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:20.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.305502+0000", 2023-10-20T03:35:20.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1060000000000006e-06 2023-10-20T03:35:20.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:20.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.306810+0000", 2023-10-20T03:35:20.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0013072419999999999 2023-10-20T03:35:20.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:20.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.306846+0000", 2023-10-20T03:35:20.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6653000000000003e-05 2023-10-20T03:35:20.226 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.227 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:20.227 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.227 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpRead(3.es6 37/31 ECSubRead(tid=28, to_read={3:7dffa048:::smithi17039300-5:head=0,4096,0}, subchunks={3:7dffa048:::smithi17039300-5:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:20.227 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:19.306191+0000", 2023-10-20T03:35:20.228 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.81978258999999998, 2023-10-20T03:35:20.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020775000000000001, 2023-10-20T03:35:20.228 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:20.228 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:20.228 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:20.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:20.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.306191+0000", 2023-10-20T03:35:20.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:20.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.306191+0000", 2023-10-20T03:35:20.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:20.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:20.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.306192+0000", 2023-10-20T03:35:20.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8999999999999995e-07 2023-10-20T03:35:20.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:20.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.306203+0000", 2023-10-20T03:35:20.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1579999999999999e-05 2023-10-20T03:35:20.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:20.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.306221+0000", 2023-10-20T03:35:20.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7227999999999998e-05 2023-10-20T03:35:20.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.233 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:20.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.306230+0000", 2023-10-20T03:35:20.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0929999999999992e-06 2023-10-20T03:35:20.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:20.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.306296+0000", 2023-10-20T03:35:20.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6778000000000003e-05 2023-10-20T03:35:20.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:20.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:20.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:20.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.306399+0000", 2023-10-20T03:35:20.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000102181 2023-10-20T03:35:20.235 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.235 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:20.236 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.236 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:20.236 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:20.236 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:20.237 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-10-20T03:35:20.238 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:20.234+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:20.260 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2023-10-20T03:35:20.270 INFO:tasks.rados.rados.0.smithi170.stdout:84: done (2 left) 2023-10-20T03:35:20.270 INFO:tasks.rados.rados.0.smithi170.stdout:85: delete oid 12 current snap is 3 2023-10-20T03:35:20.270 INFO:tasks.rados.rados.0.smithi170.stdout:83: expect (ObjNum 25 snap 0 seq_num 25) 2023-10-20T03:35:20.288 INFO:tasks.rados.rados.0.smithi170.stdout:82: finishing write tid 1 to smithi17039300-5 2023-10-20T03:35:20.288 INFO:tasks.rados.rados.0.smithi170.stdout:82: finishing write tid 2 to smithi17039300-5 2023-10-20T03:35:20.288 INFO:tasks.rados.rados.0.smithi170.stdout:82: finishing write tid 3 to smithi17039300-5 2023-10-20T03:35:20.289 INFO:tasks.rados.rados.0.smithi170.stdout:82: finishing write tid 4 to smithi17039300-5 2023-10-20T03:35:20.289 INFO:tasks.rados.rados.0.smithi170.stdout:82: oid 5 updating version 0 to 16 2023-10-20T03:35:20.289 INFO:tasks.rados.rados.0.smithi170.stdout:82: oid 5 version 16 is already newer than 14 2023-10-20T03:35:20.289 INFO:tasks.rados.rados.0.smithi170.stdout:82: oid 5 version 16 is already newer than 15 2023-10-20T03:35:20.289 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 16 (ObjNum 63 snap 3 seq_num 63) dirty exists 2023-10-20T03:35:20.290 INFO:tasks.rados.rados.0.smithi170.stdout:82: left oid 5 (ObjNum 63 snap 3 seq_num 63) 2023-10-20T03:35:20.290 INFO:tasks.rados.rados.0.smithi170.stdout:82: done (2 left) 2023-10-20T03:35:20.290 INFO:tasks.rados.rados.0.smithi170.stdout:83: done (1 left) 2023-10-20T03:35:20.290 INFO:tasks.rados.rados.0.smithi170.stdout:85: done (0 left) 2023-10-20T03:35:20.290 INFO:tasks.rados.rados.0.smithi170.stdout:86: read oid 10 snap 0 2023-10-20T03:35:20.290 INFO:tasks.rados.rados.0.smithi170.stdout:86: expect deleted 2023-10-20T03:35:20.291 INFO:tasks.rados.rados.0.smithi170.stdout:87: snap_create 2023-10-20T03:35:20.338 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:20.334+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:20.349 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:35:20.409 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:20.409 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:35:20.409 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:20.423 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-10-20T03:35:20.439 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:20.437+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:20.502 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:20.502 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:35:20.520 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:20.520 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:35:20.540 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:20.537+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:20.540 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:20.541 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:35:20.541 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:20.542 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:20.542 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:35:20.542 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:35:20.542 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:20.555 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-10-20T03:35:20.562 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:20.562 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:35:20.641 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:20.637+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:20.683 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:20.683 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:35:20.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:35:20.684 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2023-10-20T03:35:20.684 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:20.699 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-10-20T03:35:20.741 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:20.741+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:20.788 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:20.789 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:35:20.819 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:20.819 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:35:20.820 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:20.842 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:20.838+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:20.852 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:20.852 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:35:20.876 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:20.876 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:20.877 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:35:20.877 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:20.884 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:20.884 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:35:20.891 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-10-20T03:35:20.913 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:20.913 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-10-20T03:35:20.943 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:20.944 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:20.942+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:20.944 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:35:20.973 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:20.973 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:35:20.990 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:20.991 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:35:21.010 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:21.010 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:35:21.018 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:21.018 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:21.032 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-10-20T03:35:21.034 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:21.034 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-10-20T03:35:21.044 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:21.040+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:21.058 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:21.059 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:35:21.063 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:21.063 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:21.144 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:21.141+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:21.150 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:21.150 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:21.164 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-10-20T03:35:21.246 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:21.242+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:21.275 INFO:tasks.rados.rados.0.smithi170.stdout:86: done (1 left) 2023-10-20T03:35:21.276 INFO:tasks.rados.rados.0.smithi170.stdout:87: done (0 left) 2023-10-20T03:35:21.276 INFO:tasks.rados.rados.0.smithi170.stdout:88: rmattr oid 33 current snap is 4 2023-10-20T03:35:21.276 INFO:tasks.rados.rados.0.smithi170.stdout:88: done (0 left) 2023-10-20T03:35:21.276 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 32 current snap is 4 2023-10-20T03:35:21.277 INFO:tasks.rados.rados.0.smithi170.stdout:89: seq_num 65 ranges {524288=720896,1245184=376832} 2023-10-20T03:35:21.296 INFO:tasks.rados.rados.0.smithi170.stdout:89: writing smithi17039300-32 from 524288 to 1245184 tid 1 2023-10-20T03:35:21.302 INFO:tasks.rados.rados.0.smithi170.stdout:89: writing smithi17039300-32 from 1245184 to 1622016 tid 2 2023-10-20T03:35:21.302 INFO:tasks.rados.rados.0.smithi170.stdout:90: copy_from oid 45 from oid 19 current snap is 4 2023-10-20T03:35:21.302 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 7 current snap is 4 2023-10-20T03:35:21.303 INFO:tasks.rados.rados.0.smithi170.stdout:91: seq_num 66 ranges {0=737280,737280=376832} 2023-10-20T03:35:21.304 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:21.305 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:21.315 INFO:tasks.rados.rados.0.smithi170.stdout:91: writing smithi17039300-7 from 0 to 737280 tid 1 2023-10-20T03:35:21.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-10-20T03:35:21.321 INFO:tasks.rados.rados.0.smithi170.stdout:91: writing smithi17039300-7 from 737280 to 1114112 tid 2 2023-10-20T03:35:21.321 INFO:tasks.rados.rados.0.smithi170.stdout:92: copy_from oid 43 from oid 48 current snap is 4 2023-10-20T03:35:21.321 INFO:tasks.rados.rados.0.smithi170.stdout:93: delete oid 20 current snap is 4 2023-10-20T03:35:21.335 INFO:tasks.rados.rados.0.smithi170.stdout:89: finishing write tid 1 to smithi17039300-32 2023-10-20T03:35:21.337 INFO:tasks.rados.rados.0.smithi170.stdout:90: finishing copy_from to smithi17039300-45 2023-10-20T03:35:21.337 INFO:tasks.rados.rados.0.smithi170.stdout:90: got expected ENOENT (src dne) 2023-10-20T03:35:21.337 INFO:tasks.rados.rados.0.smithi170.stdout:93: done (4 left) 2023-10-20T03:35:21.337 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 4 current snap is 4 2023-10-20T03:35:21.337 INFO:tasks.rados.rados.0.smithi170.stdout:94: seq_num 68 ranges {65536=770048,835584=376832} 2023-10-20T03:35:21.347 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:21.346+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:21.348 INFO:tasks.rados.rados.0.smithi170.stdout:94: writing smithi17039300-4 from 65536 to 835584 tid 1 2023-10-20T03:35:21.355 INFO:tasks.rados.rados.0.smithi170.stdout:94: writing smithi17039300-4 from 835584 to 1212416 tid 2 2023-10-20T03:35:21.355 INFO:tasks.rados.rados.0.smithi170.stdout:89: finishing write tid 2 to smithi17039300-32 2023-10-20T03:35:21.355 INFO:tasks.rados.rados.0.smithi170.stdout:90: finishing copy_from racing read to smithi17039300-45 2023-10-20T03:35:21.355 INFO:tasks.rados.rados.0.smithi170.stdout:89: finishing write tid 3 to smithi17039300-32 2023-10-20T03:35:21.356 INFO:tasks.rados.rados.0.smithi170.stdout:89: finishing write tid 4 to smithi17039300-32 2023-10-20T03:35:21.356 INFO:tasks.rados.rados.0.smithi170.stdout:89: oid 32 updating version 0 to 6 2023-10-20T03:35:21.356 INFO:tasks.rados.rados.0.smithi170.stdout:89: oid 32 updating version 6 to 8 2023-10-20T03:35:21.356 INFO:tasks.rados.rados.0.smithi170.stdout:89: oid 32 version 8 is already newer than 7 2023-10-20T03:35:21.356 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 8 (ObjNum 65 snap 4 seq_num 65) dirty exists 2023-10-20T03:35:21.357 INFO:tasks.rados.rados.0.smithi170.stdout:89: left oid 32 (ObjNum 65 snap 4 seq_num 65) 2023-10-20T03:35:21.357 INFO:tasks.rados.rados.0.smithi170.stdout:91: finishing write tid 1 to smithi17039300-7 2023-10-20T03:35:21.357 INFO:tasks.rados.rados.0.smithi170.stdout:89: done (4 left) 2023-10-20T03:35:21.357 INFO:tasks.rados.rados.0.smithi170.stdout:90: done (3 left) 2023-10-20T03:35:21.357 INFO:tasks.rados.rados.0.smithi170.stdout:95: rollback oid 14 current snap is 4 2023-10-20T03:35:21.357 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 14 to 1 2023-10-20T03:35:21.358 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 10 current snap is 4 2023-10-20T03:35:21.358 INFO:tasks.rados.rados.0.smithi170.stdout:96: seq_num 69 ranges {0=786432,786432=376832} 2023-10-20T03:35:21.364 INFO:tasks.rados.rados.0.smithi170.stdout:96: writing smithi17039300-10 from 0 to 786432 tid 1 2023-10-20T03:35:21.369 INFO:tasks.rados.rados.0.smithi170.stdout:96: writing smithi17039300-10 from 786432 to 1163264 tid 2 2023-10-20T03:35:21.369 INFO:tasks.rados.rados.0.smithi170.stdout:91: finishing write tid 2 to smithi17039300-7 2023-10-20T03:35:21.370 INFO:tasks.rados.rados.0.smithi170.stdout:91: finishing write tid 3 to smithi17039300-7 2023-10-20T03:35:21.370 INFO:tasks.rados.rados.0.smithi170.stdout:92: finishing copy_from racing read to smithi17039300-43 2023-10-20T03:35:21.370 INFO:tasks.rados.rados.0.smithi170.stdout:92: finishing copy_from to smithi17039300-43 2023-10-20T03:35:21.370 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 43 v 5 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-10-20T03:35:21.370 INFO:tasks.rados.rados.0.smithi170.stdout:92: done (4 left) 2023-10-20T03:35:21.370 INFO:tasks.rados.rados.0.smithi170.stdout:97: rmattr oid 31 current snap is 4 2023-10-20T03:35:21.371 INFO:tasks.rados.rados.0.smithi170.stdout:97: done (4 left) 2023-10-20T03:35:21.371 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 11 current snap is 4 2023-10-20T03:35:21.371 INFO:tasks.rados.rados.0.smithi170.stdout:98: seq_num 70 ranges {180224=802816,983040=376832} 2023-10-20T03:35:21.379 INFO:tasks.rados.rados.0.smithi170.stdout:98: writing smithi17039300-11 from 180224 to 983040 tid 1 2023-10-20T03:35:21.383 INFO:tasks.rados.rados.0.smithi170.stdout:98: writing smithi17039300-11 from 983040 to 1359872 tid 2 2023-10-20T03:35:21.383 INFO:tasks.rados.rados.0.smithi170.stdout:99: read oid 24 snap -1 2023-10-20T03:35:21.384 INFO:tasks.rados.rados.0.smithi170.stdout:99: expect (ObjNum 23 snap 0 seq_num 23) 2023-10-20T03:35:21.384 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 47 current snap is 4 2023-10-20T03:35:21.384 INFO:tasks.rados.rados.0.smithi170.stdout:95: finishing rollback tid 0 to smithi17039300-14 2023-10-20T03:35:21.384 INFO:tasks.rados.rados.0.smithi170.stdout:91: finishing write tid 4 to smithi17039300-7 2023-10-20T03:35:21.384 INFO:tasks.rados.rados.0.smithi170.stdout:91: oid 7 updating version 0 to 17 2023-10-20T03:35:21.385 INFO:tasks.rados.rados.0.smithi170.stdout:91: oid 7 updating version 17 to 18 2023-10-20T03:35:21.385 INFO:tasks.rados.rados.0.smithi170.stdout:91: oid 7 updating version 18 to 19 2023-10-20T03:35:21.385 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 7 v 19 (ObjNum 66 snap 4 seq_num 66) dirty exists 2023-10-20T03:35:21.385 INFO:tasks.rados.rados.0.smithi170.stdout:91: left oid 7 (ObjNum 66 snap 4 seq_num 66) 2023-10-20T03:35:21.385 INFO:tasks.rados.rados.0.smithi170.stdout:95: finishing rollback tid 1 to smithi17039300-14 2023-10-20T03:35:21.385 INFO:tasks.rados.rados.0.smithi170.stdout:100: seq_num 71 ranges {0=425984,425984=573440,999424=196608} 2023-10-20T03:35:21.387 INFO:tasks.rados.rados.0.smithi170.stdout:100: writing smithi17039300-47 from 0 to 425984 tid 1 2023-10-20T03:35:21.393 INFO:tasks.rados.rados.0.smithi170.stdout:100: writing smithi17039300-47 from 425984 to 999424 tid 2 2023-10-20T03:35:21.396 INFO:tasks.rados.rados.0.smithi170.stdout:100: writing smithi17039300-47 from 999424 to 1196032 tid 3 2023-10-20T03:35:21.396 INFO:tasks.rados.rados.0.smithi170.stdout:91: done (6 left) 2023-10-20T03:35:21.397 INFO:tasks.rados.rados.0.smithi170.stdout:101: delete oid 18 current snap is 4 2023-10-20T03:35:21.397 INFO:tasks.rados.rados.0.smithi170.stdout:95: finishing rollback tid 2 to smithi17039300-14 2023-10-20T03:35:21.397 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 22 (ObjNum 13 snap 0 seq_num 13) dirty exists 2023-10-20T03:35:21.397 INFO:tasks.rados.rados.0.smithi170.stdout:99: expect (ObjNum 23 snap 0 seq_num 23) 2023-10-20T03:35:21.431 INFO:tasks.rados.rados.0.smithi170.stdout:95: done (6 left) 2023-10-20T03:35:21.431 INFO:tasks.rados.rados.0.smithi170.stdout:99: done (5 left) 2023-10-20T03:35:21.432 INFO:tasks.rados.rados.0.smithi170.stdout:101: done (4 left) 2023-10-20T03:35:21.432 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 36 current snap is 4 2023-10-20T03:35:21.432 INFO:tasks.rados.rados.0.smithi170.stdout:98: finishing write tid 1 to smithi17039300-11 2023-10-20T03:35:21.432 INFO:tasks.rados.rados.0.smithi170.stdout:98: finishing write tid 2 to smithi17039300-11 2023-10-20T03:35:21.433 INFO:tasks.rados.rados.0.smithi170.stdout:98: finishing write tid 3 to smithi17039300-11 2023-10-20T03:35:21.433 INFO:tasks.rados.rados.0.smithi170.stdout:94: finishing write tid 1 to smithi17039300-4 2023-10-20T03:35:21.433 INFO:tasks.rados.rados.0.smithi170.stdout:94: finishing write tid 2 to smithi17039300-4 2023-10-20T03:35:21.433 INFO:tasks.rados.rados.0.smithi170.stdout:94: finishing write tid 3 to smithi17039300-4 2023-10-20T03:35:21.433 INFO:tasks.rados.rados.0.smithi170.stdout:100: finishing write tid 1 to smithi17039300-47 2023-10-20T03:35:21.433 INFO:tasks.rados.rados.0.smithi170.stdout:100: finishing write tid 2 to smithi17039300-47 2023-10-20T03:35:21.434 INFO:tasks.rados.rados.0.smithi170.stdout:96: finishing write tid 1 to smithi17039300-10 2023-10-20T03:35:21.434 INFO:tasks.rados.rados.0.smithi170.stdout:100: finishing write tid 3 to smithi17039300-47 2023-10-20T03:35:21.434 INFO:tasks.rados.rados.0.smithi170.stdout:100: finishing write tid 4 to smithi17039300-47 2023-10-20T03:35:21.435 INFO:tasks.rados.rados.0.smithi170.stdout:96: finishing write tid 2 to smithi17039300-10 2023-10-20T03:35:21.435 INFO:tasks.rados.rados.0.smithi170.stdout:102: seq_num 73 ranges {589824=458752,1048576=475136,1523712=294912} 2023-10-20T03:35:21.438 INFO:tasks.rados.rados.0.smithi170.stdout:102: writing smithi17039300-36 from 589824 to 1048576 tid 1 2023-10-20T03:35:21.441 INFO:tasks.rados.rados.0.smithi170.stdout:102: writing smithi17039300-36 from 1048576 to 1523712 tid 2 2023-10-20T03:35:21.445 INFO:tasks.rados.rados.0.smithi170.stdout:102: writing smithi17039300-36 from 1523712 to 1818624 tid 3 2023-10-20T03:35:21.445 INFO:tasks.rados.rados.0.smithi170.stdout:103: read oid 48 snap -1 2023-10-20T03:35:21.445 INFO:tasks.rados.rados.0.smithi170.stdout:103: expect (ObjNum 47 snap 0 seq_num 47) 2023-10-20T03:35:21.445 INFO:tasks.rados.rados.0.smithi170.stdout:104: setattr oid 43 current snap is 4 2023-10-20T03:35:21.446 INFO:tasks.rados.rados.0.smithi170.stdout:98: finishing write tid 4 to smithi17039300-11 2023-10-20T03:35:21.446 INFO:tasks.rados.rados.0.smithi170.stdout:98: oid 11 updating version 0 to 7 2023-10-20T03:35:21.446 INFO:tasks.rados.rados.0.smithi170.stdout:98: oid 11 version 7 is already newer than 6 2023-10-20T03:35:21.446 INFO:tasks.rados.rados.0.smithi170.stdout:98: oid 11 updating version 7 to 8 2023-10-20T03:35:21.446 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 11 v 8 (ObjNum 70 snap 4 seq_num 70) dirty exists 2023-10-20T03:35:21.446 INFO:tasks.rados.rados.0.smithi170.stdout:98: left oid 11 (ObjNum 70 snap 4 seq_num 70) 2023-10-20T03:35:21.447 INFO:tasks.rados.rados.0.smithi170.stdout:100: finishing write tid 5 to smithi17039300-47 2023-10-20T03:35:21.447 INFO:tasks.rados.rados.0.smithi170.stdout:100: oid 47 updating version 0 to 11 2023-10-20T03:35:21.447 INFO:tasks.rados.rados.0.smithi170.stdout:100: oid 47 version 11 is already newer than 10 2023-10-20T03:35:21.447 INFO:tasks.rados.rados.0.smithi170.stdout:100: oid 47 updating version 11 to 13 2023-10-20T03:35:21.447 INFO:tasks.rados.rados.0.smithi170.stdout:100: oid 47 version 13 is already newer than 12 2023-10-20T03:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 47 v 13 (ObjNum 71 snap 4 seq_num 71) dirty exists 2023-10-20T03:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:100: left oid 47 (ObjNum 71 snap 4 seq_num 71) 2023-10-20T03:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:94: finishing write tid 4 to smithi17039300-4 2023-10-20T03:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:94: oid 4 updating version 0 to 12 2023-10-20T03:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:94: oid 4 version 12 is already newer than 11 2023-10-20T03:35:21.448 INFO:tasks.rados.rados.0.smithi170.stdout:94: oid 4 updating version 12 to 13 2023-10-20T03:35:21.449 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 13 (ObjNum 68 snap 4 seq_num 68) dirty exists 2023-10-20T03:35:21.449 INFO:tasks.rados.rados.0.smithi170.stdout:94: left oid 4 (ObjNum 68 snap 4 seq_num 68) 2023-10-20T03:35:21.449 INFO:tasks.rados.rados.0.smithi170.stdout:96: finishing write tid 3 to smithi17039300-10 2023-10-20T03:35:21.449 INFO:tasks.rados.rados.0.smithi170.stdout:96: finishing write tid 4 to smithi17039300-10 2023-10-20T03:35:21.449 INFO:tasks.rados.rados.0.smithi170.stdout:96: oid 10 updating version 0 to 15 2023-10-20T03:35:21.449 INFO:tasks.rados.rados.0.smithi170.stdout:96: oid 10 updating version 15 to 16 2023-10-20T03:35:21.450 INFO:tasks.rados.rados.0.smithi170.stdout:96: oid 10 version 16 is already newer than 14 2023-10-20T03:35:21.450 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 10 v 16 (ObjNum 69 snap 4 seq_num 69) dirty exists 2023-10-20T03:35:21.450 INFO:tasks.rados.rados.0.smithi170.stdout:96: left oid 10 (ObjNum 69 snap 4 seq_num 69) 2023-10-20T03:35:21.451 INFO:tasks.rados.rados.0.smithi170.stdout:94: done (6 left) 2023-10-20T03:35:21.451 INFO:tasks.rados.rados.0.smithi170.stdout:96: done (5 left) 2023-10-20T03:35:21.451 INFO:tasks.rados.rados.0.smithi170.stdout:98: done (4 left) 2023-10-20T03:35:21.451 INFO:tasks.rados.rados.0.smithi170.stdout:100: done (3 left) 2023-10-20T03:35:21.451 INFO:tasks.rados.rados.0.smithi170.stdout:105: delete oid 47 current snap is 4 2023-10-20T03:35:21.452 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:21.448+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:21.453 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 43 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-10-20T03:35:21.454 INFO:tasks.rados.rados.0.smithi170.stdout:103: expect (ObjNum 47 snap 0 seq_num 47) 2023-10-20T03:35:21.461 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:21.461 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:21.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-10-20T03:35:21.491 INFO:tasks.rados.rados.0.smithi170.stdout:102: finishing write tid 1 to smithi17039300-36 2023-10-20T03:35:21.491 INFO:tasks.rados.rados.0.smithi170.stdout:102: finishing write tid 2 to smithi17039300-36 2023-10-20T03:35:21.491 INFO:tasks.rados.rados.0.smithi170.stdout:102: finishing write tid 3 to smithi17039300-36 2023-10-20T03:35:21.491 INFO:tasks.rados.rados.0.smithi170.stdout:102: finishing write tid 4 to smithi17039300-36 2023-10-20T03:35:21.492 INFO:tasks.rados.rados.0.smithi170.stdout:102: finishing write tid 5 to smithi17039300-36 2023-10-20T03:35:21.492 INFO:tasks.rados.rados.0.smithi170.stdout:102: oid 36 updating version 0 to 20 2023-10-20T03:35:21.492 INFO:tasks.rados.rados.0.smithi170.stdout:102: oid 36 version 20 is already newer than 18 2023-10-20T03:35:21.492 INFO:tasks.rados.rados.0.smithi170.stdout:102: oid 36 version 20 is already newer than 19 2023-10-20T03:35:21.492 INFO:tasks.rados.rados.0.smithi170.stdout:102: oid 36 version 20 is already newer than 17 2023-10-20T03:35:21.492 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 36 v 20 (ObjNum 73 snap 4 seq_num 73) dirty exists 2023-10-20T03:35:21.493 INFO:tasks.rados.rados.0.smithi170.stdout:102: left oid 36 (ObjNum 73 snap 4 seq_num 73) 2023-10-20T03:35:21.493 INFO:tasks.rados.rados.0.smithi170.stdout:102: done (3 left) 2023-10-20T03:35:21.493 INFO:tasks.rados.rados.0.smithi170.stdout:103: done (2 left) 2023-10-20T03:35:21.493 INFO:tasks.rados.rados.0.smithi170.stdout:104: done (1 left) 2023-10-20T03:35:21.493 INFO:tasks.rados.rados.0.smithi170.stdout:105: done (0 left) 2023-10-20T03:35:21.494 INFO:tasks.rados.rados.0.smithi170.stdout:106: rollback oid 13 current snap is 4 2023-10-20T03:35:21.494 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 13 to 3 2023-10-20T03:35:21.494 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 35 current snap is 4 2023-10-20T03:35:21.494 INFO:tasks.rados.rados.0.smithi170.stdout:107: seq_num 75 ranges {573440=491520,1064960=770048} 2023-10-20T03:35:21.498 INFO:tasks.rados.rados.0.smithi170.stdout:107: writing smithi17039300-35 from 573440 to 1064960 tid 1 2023-10-20T03:35:21.504 INFO:tasks.rados.rados.0.smithi170.stdout:107: writing smithi17039300-35 from 1064960 to 1835008 tid 2 2023-10-20T03:35:21.504 INFO:tasks.rados.rados.0.smithi170.stdout:108: snap_create 2023-10-20T03:35:21.504 INFO:tasks.rados.rados.0.smithi170.stdout:106: finishing rollback tid 0 to smithi17039300-13 2023-10-20T03:35:21.514 INFO:tasks.rados.rados.0.smithi170.stdout:106: finishing rollback tid 1 to smithi17039300-13 2023-10-20T03:35:21.516 INFO:tasks.rados.rados.0.smithi170.stdout:106: finishing rollback tid 2 to smithi17039300-13 2023-10-20T03:35:21.516 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 13 v 8 (ObjNum 12 snap 0 seq_num 12) dirty exists 2023-10-20T03:35:21.533 INFO:tasks.rados.rados.0.smithi170.stdout:107: finishing write tid 1 to smithi17039300-35 2023-10-20T03:35:21.537 INFO:tasks.rados.rados.0.smithi170.stdout:107: finishing write tid 2 to smithi17039300-35 2023-10-20T03:35:21.537 INFO:tasks.rados.rados.0.smithi170.stdout:107: finishing write tid 3 to smithi17039300-35 2023-10-20T03:35:21.540 INFO:tasks.rados.rados.0.smithi170.stdout:107: finishing write tid 4 to smithi17039300-35 2023-10-20T03:35:21.540 INFO:tasks.rados.rados.0.smithi170.stdout:107: oid 35 updating version 0 to 10 2023-10-20T03:35:21.540 INFO:tasks.rados.rados.0.smithi170.stdout:107: oid 35 updating version 10 to 12 2023-10-20T03:35:21.540 INFO:tasks.rados.rados.0.smithi170.stdout:107: oid 35 version 12 is already newer than 11 2023-10-20T03:35:21.541 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 12 (ObjNum 75 snap 4 seq_num 75) dirty exists 2023-10-20T03:35:21.541 INFO:tasks.rados.rados.0.smithi170.stdout:107: left oid 35 (ObjNum 75 snap 4 seq_num 75) 2023-10-20T03:35:21.572 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:21.568+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:21.650 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:21.650 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:21.653 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:21.650+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:21.665 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-10-20T03:35:21.753 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:21.750+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:21.834 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:21.835 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:21.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-10-20T03:35:21.854 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:21.853+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:21.955 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:21.953+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:21.967 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:21.967 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:21.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-10-20T03:35:22.056 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:22.053+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:22.108 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:22.108 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:22.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-10-20T03:35:22.157 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:22.154+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:22.242 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:22.243 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:22.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-10-20T03:35:22.258 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:22.257+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:22.280 INFO:tasks.rados.rados.0.smithi170.stdout:106: done (2 left) 2023-10-20T03:35:22.280 INFO:tasks.rados.rados.0.smithi170.stdout:107: done (1 left) 2023-10-20T03:35:22.280 INFO:tasks.rados.rados.0.smithi170.stdout:108: done (0 left) 2023-10-20T03:35:22.280 INFO:tasks.rados.rados.0.smithi170.stdout:109: read oid 35 snap -1 2023-10-20T03:35:22.281 INFO:tasks.rados.rados.0.smithi170.stdout:109: expect (ObjNum 75 snap 4 seq_num 75) 2023-10-20T03:35:22.281 INFO:tasks.rados.rados.0.smithi170.stdout:110: read oid 40 snap -1 2023-10-20T03:35:22.281 INFO:tasks.rados.rados.0.smithi170.stdout:110: expect (ObjNum 39 snap 0 seq_num 39) 2023-10-20T03:35:22.281 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 13 current snap is 5 2023-10-20T03:35:22.281 INFO:tasks.rados.rados.0.smithi170.stdout:111: seq_num 76 ranges {212992=507904,720896=507904,1228800=262144} 2023-10-20T03:35:22.289 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-10-20T03:35:22.291 INFO:tasks.rados.rados.0.smithi170.stdout:111: writing smithi17039300-13 from 212992 to 720896 tid 1 2023-10-20T03:35:22.299 INFO:tasks.rados.rados.0.smithi170.stdout:111: writing smithi17039300-13 from 720896 to 1228800 tid 2 2023-10-20T03:35:22.303 INFO:tasks.rados.rados.0.smithi170.stdout:111: writing smithi17039300-13 from 1228800 to 1490944 tid 3 2023-10-20T03:35:22.303 INFO:tasks.rados.rados.0.smithi170.stdout:112: snap_remove snap 4 2023-10-20T03:35:22.359 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:22.358+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:22.385 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:22.385 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:22.400 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-10-20T03:35:22.460 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:22.456+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:22.523 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:22.524 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:22.537 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-10-20T03:35:22.561 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:22.557+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:22.661 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:22.657+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:22.667 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:22.667 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:22.681 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-10-20T03:35:22.763 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:22.756+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:22.789 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [6] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 10, 14, 3, 7, 11, 15] 2023-10-20T03:35:22.790 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-10-20T03:35:22.790 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2023-10-20T03:35:22.790 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 16 2023-10-20T03:35:22.791 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T03:35:22.798 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:22.798 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:22.812 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-10-20T03:35:22.863 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:22.862+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:22.940 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:22.941 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:22.955 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-10-20T03:35:22.963 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:22.961+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:23.064 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:35:23.065 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:23.062+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:23.109 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:35:23.109 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-10-20T03:35:23.120 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:23.120 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:23.127 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-10-20T03:35:22.706153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23986720,"num_objects":54,"num_object_clones":7,"num_object_copies":368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":3,"num_read":278,"num_read_kb":9973,"num_write":353,"num_write_kb":29252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":297,"ondisk_log_size":297,"up":129,"acting":129,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":476436,"kb_used_data":44016,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509473004,"statfs":{"total":1546188226560,"available":1545700356096,"internally_reserved":0,"allocated":45072384,"data_stored":48240771,"data_compressed":472297,"data_compressed_allocated":1036288,"data_compressed_original":2965504,"omap_allocated":25502,"internal_metadata":442473570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":54,"apply_latency_ms":54,"commit_latency_ns":54000000,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23396352,"num_objects":52,"num_object_clones":7,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":3,"num_read":142,"num_read_kb":9855,"num_write":150,"num_write_kb":25792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001305"},"pg_stats":[{"pgid":"3.f","version":"36'13","reported_seq":49,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-20T03:35:20.276247+0000","last_change":"2023-10-20T03:35:20.276675+0000","last_active":"2023-10-20T03:35:20.276247+0000","last_peered":"2023-10-20T03:35:20.276247+0000","last_clean":"2023-10-20T03:35:20.276247+0000","last_became_active":"2023-10-20T03:35:14.311454+0000","last_became_peered":"2023-10-20T03:35:14.311454+0000","last_unstale":"2023-10-20T03:35:20.276247+0000","last_undegraded":"2023-10-20T03:35:20.276247+0000","last_fullsized":"2023-10-20T03:35:20.276247+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:19.141477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057749899999999998,"stat_sum":{"num_bytes":2834432,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":11,"num_read_kb":6,"num_write":12,"num_write_kb":2768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,15,14,2,11,1],"acting":[0,3,15,14,2,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.e","version":"37'16","reported_seq":45,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:20.278383+0000","last_change":"2023-10-20T03:35:16.262200+0000","last_active":"2023-10-20T03:35:20.278383+0000","last_peered":"2023-10-20T03:35:20.278383+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:14.230273+0000","last_became_peered":"2023-10-20T03:35:14.230273+0000","last_unstale":"2023-10-20T03:35:20.278383+0000","last_undegraded":"2023-10-20T03:35:16.261909+0000","last_fullsized":"2023-10-20T03:35:14.212645+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:22.146406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2260992,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":14,"num_read_kb":1974,"num_write":13,"num_write_kb":2320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,2147483647,4,13,3,5],"acting":[11,14,2147483647,4,13,3,5],"avail_no_missing":["11(0)","3(5)","4(3)","5(6)","13(4)","14(1)"],"object_location_counts":[{"shards":"3(5),4(3),5(6),11(0),13(4),14(1)","objects":5}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"34'9","reported_seq":44,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-20T03:35:20.265537+0000","last_change":"2023-10-20T03:35:20.265924+0000","last_active":"2023-10-20T03:35:20.265537+0000","last_peered":"2023-10-20T03:35:20.265537+0000","last_clean":"2023-10-20T03:35:20.265537+0000","last_became_active":"2023-10-20T03:35:14.232138+0000","last_became_peered":"2023-10-20T03:35:14.232138+0000","last_unstale":"2023-10-20T03:35:20.265537+0000","last_undegraded":"2023-10-20T03:35:20.265537+0000","last_fullsized":"2023-10-20T03:35:20.265537+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:26.648332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056591900000000001,"stat_sum":{"num_bytes":1359872,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":4,"num_write":9,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,11,0,8,13,5],"acting":[12,4,11,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.c","version":"34'4","reported_seq":36,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-20T03:35:20.276036+0000","last_change":"2023-10-20T03:35:20.276424+0000","last_active":"2023-10-20T03:35:20.276036+0000","last_peered":"2023-10-20T03:35:20.276036+0000","last_clean":"2023-10-20T03:35:20.276036+0000","last_became_active":"2023-10-20T03:35:14.779254+0000","last_became_peered":"2023-10-20T03:35:14.779254+0000","last_unstale":"2023-10-20T03:35:20.276036+0000","last_undegraded":"2023-10-20T03:35:20.276036+0000","last_fullsized":"2023-10-20T03:35:20.276036+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:20.347821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057644599999999999,"stat_sum":{"num_bytes":786432,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,12,3,7,1,9],"acting":[0,13,12,3,7,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.b","version":"34'4","reported_seq":36,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-20T03:35:20.267828+0000","last_change":"2023-10-20T03:35:20.268877+0000","last_active":"2023-10-20T03:35:20.267828+0000","last_peered":"2023-10-20T03:35:20.267828+0000","last_clean":"2023-10-20T03:35:20.267828+0000","last_became_active":"2023-10-20T03:35:14.229448+0000","last_became_peered":"2023-10-20T03:35:14.229448+0000","last_unstale":"2023-10-20T03:35:20.267828+0000","last_undegraded":"2023-10-20T03:35:20.267828+0000","last_fullsized":"2023-10-20T03:35:20.267828+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:35.753946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024072120000000002,"stat_sum":{"num_bytes":589824,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,10,12,11,4,14],"acting":[13,1,10,12,11,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.a","version":"34'9","reported_seq":36,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:20.265382+0000","last_change":"2023-10-20T03:35:16.229627+0000","last_active":"2023-10-20T03:35:20.265382+0000","last_peered":"2023-10-20T03:35:20.265382+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:14.232949+0000","last_became_peered":"2023-10-20T03:35:14.232949+0000","last_unstale":"2023-10-20T03:35:20.265382+0000","last_undegraded":"2023-10-20T03:35:16.229223+0000","last_fullsized":"2023-10-20T03:35:14.205270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:39.882238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":720896,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":9,"num_write_kb":1008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,2147483647,10,3,15,8],"acting":[12,4,2147483647,10,3,15,8],"avail_no_missing":["12(0)","3(4)","4(1)","8(6)","10(3)","15(5)"],"object_location_counts":[{"shards":"3(4),4(1),8(6),10(3),12(0),15(5)","objects":3}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.9","version":"34'9","reported_seq":43,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-20T03:35:20.317342+0000","last_change":"2023-10-20T03:35:20.318381+0000","last_active":"2023-10-20T03:35:20.317342+0000","last_peered":"2023-10-20T03:35:20.317342+0000","last_clean":"2023-10-20T03:35:20.317342+0000","last_became_active":"2023-10-20T03:35:14.780099+0000","last_became_peered":"2023-10-20T03:35:14.780099+0000","last_unstale":"2023-10-20T03:35:20.317342+0000","last_undegraded":"2023-10-20T03:35:20.317342+0000","last_fullsized":"2023-10-20T03:35:20.317342+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:34.766349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015479160000000001,"stat_sum":{"num_bytes":1277952,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":9,"num_write_kb":2016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,10,11,4,9,3],"acting":[7,2,10,11,4,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.8","version":"34'4","reported_seq":29,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:20.277199+0000","last_change":"2023-10-20T03:35:16.333457+0000","last_active":"2023-10-20T03:35:20.277199+0000","last_peered":"2023-10-20T03:35:20.277199+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:14.226955+0000","last_became_peered":"2023-10-20T03:35:14.226955+0000","last_unstale":"2023-10-20T03:35:20.277199+0000","last_undegraded":"2023-10-20T03:35:16.333026+0000","last_fullsized":"2023-10-20T03:35:14.202565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:20.677793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1277952,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":1248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,1,11,3,5,0,7],"acting":[2147483647,1,11,3,5,0,7],"avail_no_missing":["1(1)","0(5)","3(3)","5(4)","7(6)","11(2)"],"object_location_counts":[{"shards":"0(5),1(1),3(3),5(4),7(6),11(2)","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":221,"reported_epoch":37,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:19.299424+0000","last_change":"2023-10-20T03:35:12.183237+0000","last_active":"2023-10-20T03:35:19.299424+0000","last_peered":"2023-10-20T03:35:19.299424+0000","last_clean":"2023-10-20T03:35:05.091025+0000","last_became_active":"2023-10-20T03:35:12.183237+0000","last_became_peered":"2023-10-20T03:35:12.183237+0000","last_unstale":"2023-10-20T03:35:19.299424+0000","last_undegraded":"2023-10-20T03:35:12.182136+0000","last_fullsized":"2023-10-20T03:35:12.181940+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_clean_scrub_stamp":"2023-10-20T03:35:01.005741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:23.418644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15],"acting":[15],"avail_no_missing":["15"],"object_location_counts":[{"shards":"15","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-20T03:35:19.277127+0000","last_change":"2023-10-20T03:35:05.117915+0000","last_active":"2023-10-20T03:35:19.277127+0000","last_peered":"2023-10-20T03:35:19.277127+0000","last_clean":"2023-10-20T03:35:19.277127+0000","last_became_active":"2023-10-20T03:35:05.117258+0000","last_became_peered":"2023-10-20T03:35:05.117258+0000","last_unstale":"2023-10-20T03:35:19.277127+0000","last_undegraded":"2023-10-20T03:35:19.277127+0000","last_fullsized":"2023-10-20T03:35:19.277127+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:32.359335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"36'15","reported_seq":51,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-20T03:35:20.280126+0000","last_change":"2023-10-20T03:35:20.280876+0000","last_active":"2023-10-20T03:35:20.280126+0000","last_peered":"2023-10-20T03:35:20.280126+0000","last_clean":"2023-10-20T03:35:20.280126+0000","last_became_active":"2023-10-20T03:35:14.221482+0000","last_became_peered":"2023-10-20T03:35:14.221482+0000","last_unstale":"2023-10-20T03:35:20.280126+0000","last_undegraded":"2023-10-20T03:35:20.280126+0000","last_fullsized":"2023-10-20T03:35:20.280126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:41.170132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001650749,"stat_sum":{"num_bytes":3588096,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":1878,"num_write":32,"num_write_kb":3504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,0,4,8,10,7],"acting":[14,3,0,4,8,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-20T03:35:19.281116+0000","last_change":"2023-10-20T03:35:05.201208+0000","last_active":"2023-10-20T03:35:19.281116+0000","last_peered":"2023-10-20T03:35:19.281116+0000","last_clean":"2023-10-20T03:35:19.281116+0000","last_became_active":"2023-10-20T03:35:05.200512+0000","last_became_peered":"2023-10-20T03:35:05.200512+0000","last_unstale":"2023-10-20T03:35:19.281116+0000","last_undegraded":"2023-10-20T03:35:19.281116+0000","last_fullsized":"2023-10-20T03:35:19.281116+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:30.668122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"34'8","reported_seq":45,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-20T03:35:20.267641+0000","last_change":"2023-10-20T03:35:20.268729+0000","last_active":"2023-10-20T03:35:20.267641+0000","last_peered":"2023-10-20T03:35:20.267641+0000","last_clean":"2023-10-20T03:35:20.267641+0000","last_became_active":"2023-10-20T03:35:14.311165+0000","last_became_peered":"2023-10-20T03:35:14.311165+0000","last_unstale":"2023-10-20T03:35:20.267641+0000","last_undegraded":"2023-10-20T03:35:20.267641+0000","last_fullsized":"2023-10-20T03:35:20.267641+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:27.725577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023218190000000001,"stat_sum":{"num_bytes":1490944,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":1349,"num_write":8,"num_write_kb":1456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,14,10,7,3,2],"acting":[13,4,14,10,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-20T03:35:19.277059+0000","last_change":"2023-10-20T03:35:05.113211+0000","last_active":"2023-10-20T03:35:19.277059+0000","last_peered":"2023-10-20T03:35:19.277059+0000","last_clean":"2023-10-20T03:35:19.277059+0000","last_became_active":"2023-10-20T03:35:05.112592+0000","last_became_peered":"2023-10-20T03:35:05.112592+0000","last_unstale":"2023-10-20T03:35:19.277059+0000","last_undegraded":"2023-10-20T03:35:19.277059+0000","last_fullsized":"2023-10-20T03:35:19.277059+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:27.383114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"34'7","reported_seq":39,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:20.317638+0000","last_change":"2023-10-20T03:35:16.236066+0000","last_active":"2023-10-20T03:35:20.317638+0000","last_peered":"2023-10-20T03:35:20.317638+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:14.779378+0000","last_became_peered":"2023-10-20T03:35:14.779378+0000","last_unstale":"2023-10-20T03:35:20.317638+0000","last_undegraded":"2023-10-20T03:35:16.227298+0000","last_fullsized":"2023-10-20T03:35:14.210627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:15.684433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1622016,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":2357,"num_write":7,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4,2,2147483647,15,9],"acting":[3,7,4,2,2147483647,15,9],"avail_no_missing":["3(0)","2(3)","4(2)","7(1)","9(6)","15(5)"],"object_location_counts":[{"shards":"2(3),3(0),4(2),7(1),9(6),15(5)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-20T03:35:19.281443+0000","last_change":"2023-10-20T03:35:05.114672+0000","last_active":"2023-10-20T03:35:19.281443+0000","last_peered":"2023-10-20T03:35:19.281443+0000","last_clean":"2023-10-20T03:35:19.281443+0000","last_became_active":"2023-10-20T03:35:05.114006+0000","last_became_peered":"2023-10-20T03:35:05.114006+0000","last_unstale":"2023-10-20T03:35:19.281443+0000","last_undegraded":"2023-10-20T03:35:19.281443+0000","last_fullsized":"2023-10-20T03:35:19.281443+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:28.429081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"34'9","reported_seq":46,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-20T03:35:20.278320+0000","last_change":"2023-10-20T03:35:20.279324+0000","last_active":"2023-10-20T03:35:20.278320+0000","last_peered":"2023-10-20T03:35:20.278320+0000","last_clean":"2023-10-20T03:35:20.278320+0000","last_became_active":"2023-10-20T03:35:14.309969+0000","last_became_peered":"2023-10-20T03:35:14.309969+0000","last_unstale":"2023-10-20T03:35:20.278320+0000","last_undegraded":"2023-10-20T03:35:20.278320+0000","last_fullsized":"2023-10-20T03:35:20.278320+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:31.680536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013202159999999999,"stat_sum":{"num_bytes":2129920,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":1013,"num_write":9,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,1,0,11,2,12],"acting":[10,8,1,0,11,2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.5","version":"34'6","reported_seq":42,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-20T03:35:20.315164+0000","last_change":"2023-10-20T03:35:20.315671+0000","last_active":"2023-10-20T03:35:20.315164+0000","last_peered":"2023-10-20T03:35:20.315164+0000","last_clean":"2023-10-20T03:35:20.315164+0000","last_became_active":"2023-10-20T03:35:14.779482+0000","last_became_peered":"2023-10-20T03:35:14.779482+0000","last_unstale":"2023-10-20T03:35:20.315164+0000","last_undegraded":"2023-10-20T03:35:20.315164+0000","last_fullsized":"2023-10-20T03:35:20.315164+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:16.901224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082039500000000002,"stat_sum":{"num_bytes":1228800,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":1251,"num_write":6,"num_write_kb":1664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,2,9,1,11],"acting":[8,13,5,2,9,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-20T03:35:19.295654+0000","last_change":"2023-10-20T03:35:05.114312+0000","last_active":"2023-10-20T03:35:19.295654+0000","last_peered":"2023-10-20T03:35:19.295654+0000","last_clean":"2023-10-20T03:35:19.295654+0000","last_became_active":"2023-10-20T03:35:05.113456+0000","last_became_peered":"2023-10-20T03:35:05.113456+0000","last_unstale":"2023-10-20T03:35:19.295654+0000","last_undegraded":"2023-10-20T03:35:19.295654+0000","last_fullsized":"2023-10-20T03:35:19.295654+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:22.341298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"34'2","reported_seq":26,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:20.277991+0000","last_change":"2023-10-20T03:35:16.272112+0000","last_active":"2023-10-20T03:35:20.277991+0000","last_peered":"2023-10-20T03:35:20.277991+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:14.778571+0000","last_became_peered":"2023-10-20T03:35:14.778571+0000","last_unstale":"2023-10-20T03:35:20.277991+0000","last_undegraded":"2023-10-20T03:35:16.266905+0000","last_fullsized":"2023-10-20T03:35:14.202765+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:40.365544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98304,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,15,9,2,2147483647,10,13],"acting":[1,15,9,2,2147483647,10,13],"avail_no_missing":["1(0)","2(3)","9(2)","10(5)","13(6)","15(1)"],"object_location_counts":[{"shards":"1(0),2(3),9(2),10(5),13(6),15(1)","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-20T03:35:19.255891+0000","last_change":"2023-10-20T03:35:05.122638+0000","last_active":"2023-10-20T03:35:19.255891+0000","last_peered":"2023-10-20T03:35:19.255891+0000","last_clean":"2023-10-20T03:35:19.255891+0000","last_became_active":"2023-10-20T03:35:05.122056+0000","last_became_peered":"2023-10-20T03:35:05.122056+0000","last_unstale":"2023-10-20T03:35:19.255891+0000","last_undegraded":"2023-10-20T03:35:19.255891+0000","last_fullsized":"2023-10-20T03:35:19.255891+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:21.001893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"38'7","reported_seq":31,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:20.267735+0000","last_change":"2023-10-20T03:35:16.265401+0000","last_active":"2023-10-20T03:35:20.267735+0000","last_peered":"2023-10-20T03:35:20.267735+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:14.228865+0000","last_became_peered":"2023-10-20T03:35:14.228865+0000","last_unstale":"2023-10-20T03:35:20.267735+0000","last_undegraded":"2023-10-20T03:35:16.263346+0000","last_fullsized":"2023-10-20T03:35:14.206401+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:21.660837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":5,"num_read_kb":2,"num_write":6,"num_write_kb":800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,2147483647,1,14,10,0],"acting":[13,11,2147483647,1,14,10,0],"avail_no_missing":["13(0)","0(6)","1(3)","10(5)","11(1)","14(4)"],"object_location_counts":[{"shards":"0(6),1(3),10(5),11(1),13(0),14(4)","objects":2}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-20T03:35:19.297438+0000","last_change":"2023-10-20T03:35:05.204140+0000","last_active":"2023-10-20T03:35:19.297438+0000","last_peered":"2023-10-20T03:35:19.297438+0000","last_clean":"2023-10-20T03:35:19.297438+0000","last_became_active":"2023-10-20T03:35:05.203380+0000","last_became_peered":"2023-10-20T03:35:05.203380+0000","last_unstale":"2023-10-20T03:35:19.297438+0000","last_undegraded":"2023-10-20T03:35:19.297438+0000","last_fullsized":"2023-10-20T03:35:19.297438+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:16.855327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"36'18","reported_seq":46,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:20.267062+0000","last_change":"2023-10-20T03:35:16.234185+0000","last_active":"2023-10-20T03:35:20.267062+0000","last_peered":"2023-10-20T03:35:20.267062+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:14.312252+0000","last_became_peered":"2023-10-20T03:35:14.312252+0000","last_unstale":"2023-10-20T03:35:20.267062+0000","last_undegraded":"2023-10-20T03:35:16.229093+0000","last_fullsized":"2023-10-20T03:35:14.206361+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:26.083255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1933312,"num_objects":6,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":16,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14,2147483647,2,12,0,15],"acting":[13,14,2147483647,2,12,0,15],"avail_no_missing":["13(0)","0(5)","2(3)","12(4)","14(1)","15(6)"],"object_location_counts":[{"shards":"0(5),2(3),12(4),13(0),14(1),15(6)","objects":6}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-20T03:35:19.281291+0000","last_change":"2023-10-20T03:35:05.199365+0000","last_active":"2023-10-20T03:35:19.281291+0000","last_peered":"2023-10-20T03:35:19.281291+0000","last_clean":"2023-10-20T03:35:19.281291+0000","last_became_active":"2023-10-20T03:35:05.198615+0000","last_became_peered":"2023-10-20T03:35:05.198615+0000","last_unstale":"2023-10-20T03:35:19.281291+0000","last_undegraded":"2023-10-20T03:35:19.281291+0000","last_fullsized":"2023-10-20T03:35:19.281291+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:23.667783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":23396352,"num_objects":52,"num_object_clones":7,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":3,"num_read":142,"num_read_kb":9855,"num_write":150,"num_write_kb":25792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":38916096,"data_stored":43847680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":112,"acting":112,"num_store_stats":15},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15078,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280519,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29840,"kb_used_data":2812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342000,"statfs":{"total":96636764160,"available":96606208000,"internally_reserved":0,"allocated":2879488,"data_stored":2850280,"data_compressed":30930,"data_compressed_allocated":65536,"data_compressed_original":131072,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]},{"osd":15,"up_from":22,"seq":94489280520,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29204,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342636,"statfs":{"total":96636764160,"available":96606859264,"internally_reserved":0,"allocated":2236416,"data_stored":2664471,"data_compressed":36482,"data_compressed_allocated":94208,"data_compressed_original":634880,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3560000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.597}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.452}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.956}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.694}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.085}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7330000000000001}]}]},{"osd":14,"up_from":22,"seq":94489280520,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30556,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341284,"statfs":{"total":96636764160,"available":96605474816,"internally_reserved":0,"allocated":3616768,"data_stored":4169192,"data_compressed":30930,"data_compressed_allocated":65536,"data_compressed_original":131072,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,7,8,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4630000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.383}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4470000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5009999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0840000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.21}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.792}]}]},{"osd":5,"up_from":21,"seq":90194313224,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28888,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342952,"statfs":{"total":96636764160,"available":96607182848,"internally_reserved":0,"allocated":1908736,"data_stored":1762221,"data_compressed":28943,"data_compressed_allocated":61440,"data_compressed_original":122880,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1830000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3080000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]}]},{"osd":0,"up_from":22,"seq":94489280519,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30852,"kb_used_data":3828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340988,"statfs":{"total":96636764160,"available":96605171712,"internally_reserved":0,"allocated":3919872,"data_stored":4451816,"data_compressed":30930,"data_compressed_allocated":65536,"data_compressed_original":131072,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93600000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.681}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]}]},{"osd":1,"up_from":22,"seq":94489280519,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29628,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342212,"statfs":{"total":96636764160,"available":96606425088,"internally_reserved":0,"allocated":2666496,"data_stored":2495991,"data_compressed":28943,"data_compressed_allocated":61440,"data_compressed_original":122880,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.534}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]}]},{"osd":2,"up_from":22,"seq":94489280519,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30472,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341368,"statfs":{"total":96636764160,"available":96605560832,"internally_reserved":0,"allocated":3534848,"data_stored":3487223,"data_compressed":28943,"data_compressed_allocated":61440,"data_compressed_original":122880,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,7,8,9,10,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.407}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.595}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.395}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.226}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0800000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.728}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2839999999999998}]}]},{"osd":3,"up_from":22,"seq":94489280519,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31284,"kb_used_data":4256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340556,"statfs":{"total":96636764160,"available":96604729344,"internally_reserved":0,"allocated":4358144,"data_stored":4787688,"data_compressed":30930,"data_compressed_allocated":65536,"data_compressed_original":131072,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,7,8,9,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.448}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1599999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.319}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]}]},{"osd":4,"up_from":22,"seq":94489280520,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30544,"kb_used_data":3524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341296,"statfs":{"total":96636764160,"available":96605487104,"internally_reserved":0,"allocated":3608576,"data_stored":4040183,"data_compressed":28943,"data_compressed_allocated":61440,"data_compressed_original":122880,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[2,3,5,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]}]},{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":323584,"data_stored":647661,"data_compressed":18691,"data_compressed_allocated":57344,"data_compressed_original":561152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,3,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4139999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7290000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7989999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.738}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9089999999999998}]}]},{"osd":7,"up_from":22,"seq":94489280518,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29848,"kb_used_data":2824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341992,"statfs":{"total":96636764160,"available":96606199808,"internally_reserved":0,"allocated":2891776,"data_stored":3302903,"data_compressed":28943,"data_compressed_allocated":61440,"data_compressed_original":122880,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,2,3,4,5,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3149999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4159999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.536}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8420000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3799999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280519,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29600,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342240,"statfs":{"total":96636764160,"available":96606453760,"internally_reserved":0,"allocated":2637824,"data_stored":3048951,"data_compressed":28943,"data_compressed_allocated":61440,"data_compressed_original":122880,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,5,7,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.329}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.855}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.962}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]}]},{"osd":9,"up_from":22,"seq":94489280519,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28636,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343204,"statfs":{"total":96636764160,"available":96607440896,"internally_reserved":0,"allocated":1646592,"data_stored":1474024,"data_compressed":30930,"data_compressed_allocated":65536,"data_compressed_original":131072,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]}]},{"osd":10,"up_from":22,"seq":94489280518,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29860,"kb_used_data":2836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341980,"statfs":{"total":96636764160,"available":96606187520,"internally_reserved":0,"allocated":2904064,"data_stored":3315191,"data_compressed":28943,"data_compressed_allocated":61440,"data_compressed_original":122880,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0940000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.276}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4379999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]}]},{"osd":11,"up_from":22,"seq":94489280520,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30604,"kb_used_data":3580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341236,"statfs":{"total":96636764160,"available":96605425664,"internally_reserved":0,"allocated":3665920,"data_stored":3519384,"data_compressed":28943,"data_compressed_allocated":61440,"data_compressed_original":122880,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4690000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5960000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.536}]}]},{"osd":12,"up_from":22,"seq":94489280518,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29248,"kb_used_data":2220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342592,"statfs":{"total":96636764160,"available":96606814208,"internally_reserved":0,"allocated":2273280,"data_stored":2223592,"data_compressed":30930,"data_compressed_allocated":65536,"data_compressed_original":131072,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,7,8,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.466}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.641}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9590000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]}]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3526656,"data_stored":4231168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2285568,"data_stored":2285568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3153920,"data_stored":3276800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3964928,"data_stored":4567040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3227648,"data_stored":3829760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1531904,"data_stored":1552384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2510848,"data_stored":3092480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2256896,"data_stored":2838528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1253376,"data_stored":1253376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2523136,"data_stored":3104768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3289088,"data_stored":3309568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1880064,"data_stored":2002944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":2486272,"data_stored":2629632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":3223552,"data_stored":3948544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1925120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T03:35:23.131 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2023-10-20T03:35:23.134 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-10-20T03:35:23.166 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:23.164+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:23.226 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:23.226 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.266 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:23.265+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:23.292 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:23.292 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:23.293 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2023-10-20T03:35:23.301 INFO:tasks.rados.rados.0.smithi170.stdout:112: done (3 left) 2023-10-20T03:35:23.301 INFO:tasks.rados.rados.0.smithi170.stdout:113: rmattr oid 17 current snap is 5 2023-10-20T03:35:23.301 INFO:tasks.rados.rados.0.smithi170.stdout:113: done (3 left) 2023-10-20T03:35:23.301 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 1 current snap is 5 2023-10-20T03:35:23.301 INFO:tasks.rados.rados.0.smithi170.stdout:114: seq_num 77 ranges {32768=524288,557056=524288,1081344=245760} 2023-10-20T03:35:23.307 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-10-20T03:35:23.309 INFO:tasks.rados.rados.0.smithi170.stdout:114: writing smithi17039300-1 from 32768 to 557056 tid 1 2023-10-20T03:35:23.318 INFO:tasks.rados.rados.0.smithi170.stdout:114: writing smithi17039300-1 from 557056 to 1081344 tid 2 2023-10-20T03:35:23.322 INFO:tasks.rados.rados.0.smithi170.stdout:114: writing smithi17039300-1 from 1081344 to 1327104 tid 3 2023-10-20T03:35:23.323 INFO:tasks.rados.rados.0.smithi170.stdout:110: expect (ObjNum 39 snap 0 seq_num 39) 2023-10-20T03:35:23.367 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:23.360+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:23.369 INFO:tasks.rados.rados.0.smithi170.stdout:109: expect (ObjNum 75 snap 4 seq_num 75) 2023-10-20T03:35:23.432 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:23.433 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:23.445 INFO:tasks.rados.rados.0.smithi170.stdout:111: finishing write tid 1 to smithi17039300-13 2023-10-20T03:35:23.445 INFO:tasks.rados.rados.0.smithi170.stdout:111: finishing write tid 2 to smithi17039300-13 2023-10-20T03:35:23.445 INFO:tasks.rados.rados.0.smithi170.stdout:111: finishing write tid 3 to smithi17039300-13 2023-10-20T03:35:23.446 INFO:tasks.rados.rados.0.smithi170.stdout:111: finishing write tid 4 to smithi17039300-13 2023-10-20T03:35:23.446 INFO:tasks.rados.rados.0.smithi170.stdout:111: finishing write tid 5 to smithi17039300-13 2023-10-20T03:35:23.446 INFO:tasks.rados.rados.0.smithi170.stdout:111: oid 13 updating version 0 to 16 2023-10-20T03:35:23.446 INFO:tasks.rados.rados.0.smithi170.stdout:111: oid 13 updating version 16 to 17 2023-10-20T03:35:23.446 INFO:tasks.rados.rados.0.smithi170.stdout:111: oid 13 version 17 is already newer than 15 2023-10-20T03:35:23.446 INFO:tasks.rados.rados.0.smithi170.stdout:111: oid 13 version 17 is already newer than 14 2023-10-20T03:35:23.447 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 13 v 17 (ObjNum 76 snap 5 seq_num 76) dirty exists 2023-10-20T03:35:23.447 INFO:tasks.rados.rados.0.smithi170.stdout:111: left oid 13 (ObjNum 76 snap 5 seq_num 76) 2023-10-20T03:35:23.447 INFO:tasks.rados.rados.0.smithi170.stdout:109: done (3 left) 2023-10-20T03:35:23.447 INFO:tasks.rados.rados.0.smithi170.stdout:110: done (2 left) 2023-10-20T03:35:23.447 INFO:tasks.rados.rados.0.smithi170.stdout:111: done (1 left) 2023-10-20T03:35:23.448 INFO:tasks.rados.rados.0.smithi170.stdout:115: copy_from oid 25 from oid 43 current snap is 5 2023-10-20T03:35:23.448 INFO:tasks.rados.rados.0.smithi170.stdout:114: finishing write tid 1 to smithi17039300-1 2023-10-20T03:35:23.448 INFO:tasks.rados.rados.0.smithi170.stdout:114: finishing write tid 2 to smithi17039300-1 2023-10-20T03:35:23.448 INFO:tasks.rados.rados.0.smithi170.stdout:114: finishing write tid 3 to smithi17039300-1 2023-10-20T03:35:23.448 INFO:tasks.rados.rados.0.smithi170.stdout:114: finishing write tid 4 to smithi17039300-1 2023-10-20T03:35:23.449 INFO:tasks.rados.rados.0.smithi170.stdout:114: finishing write tid 5 to smithi17039300-1 2023-10-20T03:35:23.449 INFO:tasks.rados.rados.0.smithi170.stdout:114: oid 1 updating version 0 to 13 2023-10-20T03:35:23.449 INFO:tasks.rados.rados.0.smithi170.stdout:114: oid 1 updating version 13 to 15 2023-10-20T03:35:23.449 INFO:tasks.rados.rados.0.smithi170.stdout:114: oid 1 updating version 15 to 16 2023-10-20T03:35:23.449 INFO:tasks.rados.rados.0.smithi170.stdout:114: oid 1 version 16 is already newer than 14 2023-10-20T03:35:23.450 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 1 v 16 (ObjNum 77 snap 5 seq_num 77) dirty exists 2023-10-20T03:35:23.450 INFO:tasks.rados.rados.0.smithi170.stdout:114: left oid 1 (ObjNum 77 snap 5 seq_num 77) 2023-10-20T03:35:23.450 INFO:tasks.rados.rados.0.smithi170.stdout:114: done (1 left) 2023-10-20T03:35:23.450 INFO:tasks.rados.rados.0.smithi170.stdout:116: read oid 6 snap -1 2023-10-20T03:35:23.451 INFO:tasks.rados.rados.0.smithi170.stdout:116: expect (ObjNum 5 snap 0 seq_num 5) 2023-10-20T03:35:23.451 INFO:tasks.rados.rados.0.smithi170.stdout:117: read oid 42 snap -1 2023-10-20T03:35:23.451 INFO:tasks.rados.rados.0.smithi170.stdout:117: expect (ObjNum 60 snap 2 seq_num 60) 2023-10-20T03:35:23.451 INFO:tasks.rados.rados.0.smithi170.stdout:118: rollback oid 9 current snap is 5 2023-10-20T03:35:23.452 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 9 to 1 2023-10-20T03:35:23.452 INFO:tasks.rados.rados.0.smithi170.stdout:119: snap_create 2023-10-20T03:35:23.453 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-10-20T03:35:23.455 INFO:tasks.rados.rados.0.smithi170.stdout:116: expect (ObjNum 5 snap 0 seq_num 5) 2023-10-20T03:35:23.458 INFO:tasks.rados.rados.0.smithi170.stdout:118: finishing rollback tid 0 to smithi17039300-9 2023-10-20T03:35:23.460 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:23.461 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.462 INFO:tasks.rados.rados.0.smithi170.stdout:118: finishing rollback tid 1 to smithi17039300-9 2023-10-20T03:35:23.463 INFO:tasks.rados.rados.0.smithi170.stdout:118: finishing rollback tid 2 to smithi17039300-9 2023-10-20T03:35:23.463 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 12 (ObjNum 8 snap 0 seq_num 8) dirty exists 2023-10-20T03:35:23.468 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:23.465+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:23.473 INFO:tasks.rados.rados.0.smithi170.stdout:117: expect (ObjNum 60 snap 2 seq_num 60) 2023-10-20T03:35:23.498 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:23.498 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.527 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:23.528 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.536 INFO:tasks.rados.rados.0.smithi170.stdout:115: finishing copy_from racing read to smithi17039300-25 2023-10-20T03:35:23.536 INFO:tasks.rados.rados.0.smithi170.stdout:115: finishing copy_from to smithi17039300-25 2023-10-20T03:35:23.536 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-10-20T03:35:23.556 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:23.556 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.569 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:23.565+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:23.578 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:23.578 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:23.585 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:23.585 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.586 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:23.592 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-10-20T03:35:23.618 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:23.618 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.643 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:23.644 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.669 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:23.668+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:23.671 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:23.671 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.700 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:23.700 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.727 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:23.728 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.754 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:23.754 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.761 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:23.761 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:23.761 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:35:23.761 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:35:23.762 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:23.770 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:23.766+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:23.775 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-10-20T03:35:23.780 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:23.780 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.807 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:23.808 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:35:23.808 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:35:23.805+0000 7feeecf26640 0 -- 172.21.15.33:0/2522291731 >> [v2:172.21.15.170:6819/826332016,v1:172.21.15.170:6822/826332016] conn(0x7feee821e790 msgr2=0x7feee8220b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-10-20T03:35:23.871 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:23.868+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:23.943 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:23.944 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2023-10-20T03:35:23.944 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2023-10-20T03:35:23.944 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2023-10-20T03:35:23.944 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.945 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:205 3.es0 3:76e5b9dc:::smithi17039300-27:head [stat,delete] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-10-20T03:35:23.945 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:18.268774+0000", 2023-10-20T03:35:23.945 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 5.6725061329999997, 2023-10-20T03:35:23.945 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012641820999999999, 2023-10-20T03:35:23.945 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:23.946 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:23.946 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:23.946 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:23.946 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:23.947 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 205 2023-10-20T03:35:23.947 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.947 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:23.947 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.947 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:23.948 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.268774+0000", 2023-10-20T03:35:23.948 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:23.948 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.948 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.948 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:23.949 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.268774+0000", 2023-10-20T03:35:23.949 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:23.949 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.949 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.949 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:23.949 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.268777+0000", 2023-10-20T03:35:23.950 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3469999999999999e-06 2023-10-20T03:35:23.950 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.950 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.950 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:23.951 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.268791+0000", 2023-10-20T03:35:23.951 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3986e-05 2023-10-20T03:35:23.951 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.951 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.951 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:23.952 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.268841+0000", 2023-10-20T03:35:23.952 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9261000000000002e-05 2023-10-20T03:35:23.952 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.952 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.952 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:23.953 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.268865+0000", 2023-10-20T03:35:23.953 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4743e-05 2023-10-20T03:35:23.953 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.953 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.953 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:23.954 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.269050+0000", 2023-10-20T03:35:23.954 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018490600000000001 2023-10-20T03:35:23.954 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.954 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.954 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:23.955 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.269330+0000", 2023-10-20T03:35:23.955 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00027926300000000003 2023-10-20T03:35:23.955 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.955 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.955 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:23.956 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.271128+0000", 2023-10-20T03:35:23.956 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017987299999999999 2023-10-20T03:35:23.956 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.956 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.956 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:23.957 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.274480+0000", 2023-10-20T03:35:23.957 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033517 2023-10-20T03:35:23.957 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.957 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.957 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:23.958 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.280926+0000", 2023-10-20T03:35:23.958 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0064456799999999996 2023-10-20T03:35:23.958 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.958 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.959 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:23.959 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.281416+0000", 2023-10-20T03:35:23.959 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00049020499999999996 2023-10-20T03:35:23.959 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:23.959 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:23.960 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:23.960 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.960 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.960 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 36/31 ECSubWriteReply(tid=21, last_complete=36'12, committed=1, applied=1))", 2023-10-20T03:35:23.960 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:18.280475+0000", 2023-10-20T03:35:23.960 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 5.6608056160000002, 2023-10-20T03:35:23.961 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001974487, 2023-10-20T03:35:23.961 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:23.961 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:23.961 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:23.961 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.962 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:23.962 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.280475+0000", 2023-10-20T03:35:23.962 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:23.962 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.962 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.963 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:23.963 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.280475+0000", 2023-10-20T03:35:23.963 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:23.963 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.963 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.963 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:23.964 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.280477+0000", 2023-10-20T03:35:23.964 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0839999999999999e-06 2023-10-20T03:35:23.964 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.964 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.964 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:23.965 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.280489+0000", 2023-10-20T03:35:23.965 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2332e-05 2023-10-20T03:35:23.965 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.965 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.965 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:23.966 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.280527+0000", 2023-10-20T03:35:23.966 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7901999999999997e-05 2023-10-20T03:35:23.966 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.966 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.966 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:23.967 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.280548+0000", 2023-10-20T03:35:23.967 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1115999999999999e-05 2023-10-20T03:35:23.967 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.967 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.967 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:23.968 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.280703+0000", 2023-10-20T03:35:23.968 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015519700000000001 2023-10-20T03:35:23.968 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.968 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.968 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:23.968 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.282449+0000", 2023-10-20T03:35:23.969 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017458560000000001 2023-10-20T03:35:23.969 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:23.969 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:23.969 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:23.969 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.970 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.970 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs5 36/31 ECSubWrite(tid=32, reqid=client.4570.0:206, at_version=36'13, trim_to=0'0, roll_forward_to=34'11))", 2023-10-20T03:35:23.970 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:18.282530+0000", 2023-10-20T03:35:23.970 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 5.6587502790000004, 2023-10-20T03:35:23.970 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0031415789999999998, 2023-10-20T03:35:23.971 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:23.971 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:23.971 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:23.971 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.972 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:23.972 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.282530+0000", 2023-10-20T03:35:23.972 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:23.972 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.972 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.282530+0000", 2023-10-20T03:35:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:23.974 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.282531+0000", 2023-10-20T03:35:23.974 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6400000000000002e-07 2023-10-20T03:35:23.974 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.974 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.974 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:23.975 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.282582+0000", 2023-10-20T03:35:23.975 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0958000000000002e-05 2023-10-20T03:35:23.975 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.975 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.976 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:23.976 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.282623+0000", 2023-10-20T03:35:23.976 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1431999999999999e-05 2023-10-20T03:35:23.976 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.976 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.977 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:23.977 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.282646+0000", 2023-10-20T03:35:23.977 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2489000000000001e-05 2023-10-20T03:35:23.977 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.978 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.978 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:23.978 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.282802+0000", 2023-10-20T03:35:23.978 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015686000000000001 2023-10-20T03:35:23.978 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.979 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.979 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:23.979 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.282827+0000", 2023-10-20T03:35:23.979 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4836000000000001e-05 2023-10-20T03:35:23.980 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.980 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.980 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:23.980 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.285583+0000", 2023-10-20T03:35:23.980 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00275567 2023-10-20T03:35:23.981 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.981 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.981 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:23.981 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.285672+0000", 2023-10-20T03:35:23.981 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.8670000000000003e-05 2023-10-20T03:35:23.982 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:23.982 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:23.982 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:23.982 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.983 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.983 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs5 36/31 ECSubWrite(tid=33, reqid=client.4570.0:206, at_version=0'0, trim_to=0'0, roll_forward_to=36'13))", 2023-10-20T03:35:23.983 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:18.289816+0000", 2023-10-20T03:35:23.983 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 5.6514642510000002, 2023-10-20T03:35:23.983 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002088786, 2023-10-20T03:35:23.984 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:23.984 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:23.984 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:23.984 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.985 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:23.985 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.289816+0000", 2023-10-20T03:35:23.985 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:23.985 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.985 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.986 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:23.986 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.289816+0000", 2023-10-20T03:35:23.986 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:23.986 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.987 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.987 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:23.987 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.289818+0000", 2023-10-20T03:35:23.987 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.863e-06 2023-10-20T03:35:23.987 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.988 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.988 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:23.988 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.289848+0000", 2023-10-20T03:35:23.988 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9845999999999999e-05 2023-10-20T03:35:23.988 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.989 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.989 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:23.989 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.289905+0000", 2023-10-20T03:35:23.989 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7268000000000001e-05 2023-10-20T03:35:23.989 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.989 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.990 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:23.990 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.289929+0000", 2023-10-20T03:35:23.990 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3598999999999999e-05 2023-10-20T03:35:23.990 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.990 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.991 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:23.991 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.290128+0000", 2023-10-20T03:35:23.991 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019931899999999999 2023-10-20T03:35:23.991 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.991 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.991 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:23.992 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.290154+0000", 2023-10-20T03:35:23.992 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5799000000000002e-05 2023-10-20T03:35:23.992 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.992 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.993 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:23.993 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.291825+0000", 2023-10-20T03:35:23.993 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001671701 2023-10-20T03:35:23.993 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.994 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.994 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:23.994 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.291905+0000", 2023-10-20T03:35:23.994 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9390999999999999e-05 2023-10-20T03:35:23.995 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:23.995 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:23.995 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:23.996 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:23.996 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:23.996 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:208 3.es0 3:7dffa048:::smithi17039300-5:head [append 0~688128] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:23.997 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.269565+0000", 2023-10-20T03:35:23.997 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6717147399999996, 2023-10-20T03:35:23.997 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029923571, 2023-10-20T03:35:23.998 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:23.998 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:23.998 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:23.999 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:23.999 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:23.999 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 208 2023-10-20T03:35:23.999 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.000 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.000 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.000 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.000 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.269565+0000", 2023-10-20T03:35:24.001 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.001 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.001 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.001 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.001 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.269565+0000", 2023-10-20T03:35:24.002 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.002 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.002 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.002 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.002 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.269569+0000", 2023-10-20T03:35:24.003 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0230000000000001e-06 2023-10-20T03:35:24.003 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.003 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.003 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.003 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270337+0000", 2023-10-20T03:35:24.004 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00076892900000000001 2023-10-20T03:35:24.004 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.004 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.004 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.004 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270372+0000", 2023-10-20T03:35:24.004 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4652e-05 2023-10-20T03:35:24.005 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.005 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.005 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.005 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270391+0000", 2023-10-20T03:35:24.005 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8819000000000001e-05 2023-10-20T03:35:24.006 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.006 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.006 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.006 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270555+0000", 2023-10-20T03:35:24.006 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016383700000000001 2023-10-20T03:35:24.007 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.007 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.007 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.007 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270730+0000", 2023-10-20T03:35:24.007 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017536 2023-10-20T03:35:24.008 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.008 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.008 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:24.008 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.272967+0000", 2023-10-20T03:35:24.008 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002236601 2023-10-20T03:35:24.008 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.009 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.009 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:24.009 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298706+0000", 2023-10-20T03:35:24.009 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025739482000000001 2023-10-20T03:35:24.009 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.010 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.010 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.010 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299042+0000", 2023-10-20T03:35:24.010 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00033608100000000001 2023-10-20T03:35:24.010 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.011 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.011 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.011 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299489+0000", 2023-10-20T03:35:24.011 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00044678699999999998 2023-10-20T03:35:24.011 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.012 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.012 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.012 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.012 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.012 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:209 3.es0 3:7dffa048:::smithi17039300-5:head [append 0~376832] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:24.012 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.277974+0000", 2023-10-20T03:35:24.013 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6633059049999996, 2023-10-20T03:35:24.013 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.023138072999999999, 2023-10-20T03:35:24.013 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.013 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.013 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:24.014 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:24.014 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:24.014 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 209 2023-10-20T03:35:24.014 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.015 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.015 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.015 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.016 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.277974+0000", 2023-10-20T03:35:24.016 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.016 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.017 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.017 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.017 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.277974+0000", 2023-10-20T03:35:24.018 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.018 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.019 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.019 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.019 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.277977+0000", 2023-10-20T03:35:24.020 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.553e-06 2023-10-20T03:35:24.020 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.020 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.021 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.021 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278381+0000", 2023-10-20T03:35:24.021 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00040455200000000003 2023-10-20T03:35:24.022 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.022 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.022 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.023 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278416+0000", 2023-10-20T03:35:24.023 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4269e-05 2023-10-20T03:35:24.023 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.023 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.024 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.024 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278440+0000", 2023-10-20T03:35:24.024 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3926000000000001e-05 2023-10-20T03:35:24.024 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.024 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.025 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.025 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.287724+0000", 2023-10-20T03:35:24.025 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0092847510000000008 2023-10-20T03:35:24.025 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.025 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.025 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.026 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.288029+0000", 2023-10-20T03:35:24.026 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00030455899999999998 2023-10-20T03:35:24.026 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.026 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.027 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:24.027 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.289828+0000", 2023-10-20T03:35:24.027 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017987210000000001 2023-10-20T03:35:24.028 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.028 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.028 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:24.028 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.300205+0000", 2023-10-20T03:35:24.029 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010377752000000001 2023-10-20T03:35:24.029 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.029 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.029 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.029 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.300337+0000", 2023-10-20T03:35:24.030 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000131755 2023-10-20T03:35:24.030 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.030 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.030 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.030 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.301112+0000", 2023-10-20T03:35:24.031 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00077523499999999999 2023-10-20T03:35:24.031 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.031 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.031 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.032 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.032 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:210 3.es0 3:7dffa048:::smithi17039300-5:head [setxattr _header (55) in=7b] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:24.032 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.278490+0000", 2023-10-20T03:35:24.032 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6627900269999998, 2023-10-20T03:35:24.032 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027549357999999999, 2023-10-20T03:35:24.033 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.033 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.033 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:24.033 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:24.033 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:24.034 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210 2023-10-20T03:35:24.034 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.034 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.034 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.034 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.034 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278490+0000", 2023-10-20T03:35:24.035 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.035 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.035 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.035 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.036 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278490+0000", 2023-10-20T03:35:24.036 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.037 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.037 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278492+0000", 2023-10-20T03:35:24.037 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.59e-06 2023-10-20T03:35:24.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.037 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.038 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.038 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278501+0000", 2023-10-20T03:35:24.038 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.986e-06 2023-10-20T03:35:24.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.039 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.039 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.039 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278519+0000", 2023-10-20T03:35:24.039 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8238999999999999e-05 2023-10-20T03:35:24.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.040 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.040 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.040 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278536+0000", 2023-10-20T03:35:24.040 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7376000000000001e-05 2023-10-20T03:35:24.040 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.040 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.041 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.041 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.301212+0000", 2023-10-20T03:35:24.041 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.022675563999999999 2023-10-20T03:35:24.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.041 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.041 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.042 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.301368+0000", 2023-10-20T03:35:24.042 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015616 2023-10-20T03:35:24.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.042 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.043 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:24.043 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.302117+0000", 2023-10-20T03:35:24.043 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00074924300000000002 2023-10-20T03:35:24.043 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.043 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.044 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:24.044 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.303929+0000", 2023-10-20T03:35:24.044 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0018115830000000001 2023-10-20T03:35:24.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.045 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.045 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.045 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.305714+0000", 2023-10-20T03:35:24.045 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001785502 2023-10-20T03:35:24.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.046 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.046 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.046 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.306040+0000", 2023-10-20T03:35:24.046 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00032511500000000001 2023-10-20T03:35:24.046 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.047 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.047 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.047 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.047 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:211 3.es0 3:7dffa048:::smithi17039300-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:24.048 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.278563+0000", 2023-10-20T03:35:24.048 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6627177199999998, 2023-10-20T03:35:24.048 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030205164, 2023-10-20T03:35:24.048 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.048 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:24.049 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:24.049 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:24.049 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:24.049 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211 2023-10-20T03:35:24.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.050 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.050 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.050 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.050 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278563+0000", 2023-10-20T03:35:24.051 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.051 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.051 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.051 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278563+0000", 2023-10-20T03:35:24.052 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.052 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.052 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.052 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278564+0000", 2023-10-20T03:35:24.053 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.826e-06 2023-10-20T03:35:24.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.053 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.053 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.053 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278572+0000", 2023-10-20T03:35:24.054 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0299999999999994e-06 2023-10-20T03:35:24.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.054 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.054 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.054 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278598+0000", 2023-10-20T03:35:24.055 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5593e-05 2023-10-20T03:35:24.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.055 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.055 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.055 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278619+0000", 2023-10-20T03:35:24.056 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0755e-05 2023-10-20T03:35:24.056 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.056 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.056 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.056 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.302759+0000", 2023-10-20T03:35:24.057 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.024140663 2023-10-20T03:35:24.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.057 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.057 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:24.057 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.303005+0000", 2023-10-20T03:35:24.057 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024529499999999997 2023-10-20T03:35:24.058 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.058 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.058 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.058 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.306691+0000", 2023-10-20T03:35:24.058 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0036863809999999999 2023-10-20T03:35:24.059 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.059 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.059 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.059 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.306820+0000", 2023-10-20T03:35:24.060 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012892200000000001 2023-10-20T03:35:24.060 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.060 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.060 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.060 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.308768+0000", 2023-10-20T03:35:24.061 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001947699 2023-10-20T03:35:24.061 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.061 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.061 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.062 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.062 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=23, last_complete=37'14, committed=1, applied=1))", 2023-10-20T03:35:24.062 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.282066+0000", 2023-10-20T03:35:24.062 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6592140930000001, 2023-10-20T03:35:24.063 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.01609031, 2023-10-20T03:35:24.063 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.063 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:24.063 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.063 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.064 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.064 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282066+0000", 2023-10-20T03:35:24.064 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.064 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.064 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.064 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.065 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282066+0000", 2023-10-20T03:35:24.065 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.065 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.065 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.065 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.066 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282074+0000", 2023-10-20T03:35:24.066 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9840000000000001e-06 2023-10-20T03:35:24.066 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.066 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.066 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.066 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282084+0000", 2023-10-20T03:35:24.067 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0020000000000001e-05 2023-10-20T03:35:24.067 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.067 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.067 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.067 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282113+0000", 2023-10-20T03:35:24.068 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8572e-05 2023-10-20T03:35:24.068 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.068 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.068 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.068 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282134+0000", 2023-10-20T03:35:24.069 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1622000000000001e-05 2023-10-20T03:35:24.069 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.069 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.069 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.069 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298102+0000", 2023-10-20T03:35:24.069 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015968110000000001 2023-10-20T03:35:24.070 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.070 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.070 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.071 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298156+0000", 2023-10-20T03:35:24.071 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4002000000000002e-05 2023-10-20T03:35:24.071 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.071 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.071 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.072 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.072 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.072 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=23, last_complete=37'14, committed=1, applied=1))", 2023-10-20T03:35:24.072 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.282628+0000", 2023-10-20T03:35:24.072 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6586519260000001, 2023-10-20T03:35:24.072 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015776847, 2023-10-20T03:35:24.073 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.073 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:24.073 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.073 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.073 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.074 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282628+0000", 2023-10-20T03:35:24.074 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.074 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.074 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.074 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.075 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282628+0000", 2023-10-20T03:35:24.075 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.075 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.075 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.075 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.075 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282630+0000", 2023-10-20T03:35:24.076 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9199999999999998e-06 2023-10-20T03:35:24.076 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.076 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.076 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.076 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282640+0000", 2023-10-20T03:35:24.077 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0056e-05 2023-10-20T03:35:24.077 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.077 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.077 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.078 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282670+0000", 2023-10-20T03:35:24.078 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0114999999999999e-05 2023-10-20T03:35:24.078 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.078 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.078 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.079 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282694+0000", 2023-10-20T03:35:24.079 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3623000000000001e-05 2023-10-20T03:35:24.079 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.079 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.079 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.080 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298309+0000", 2023-10-20T03:35:24.080 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015614663000000001 2023-10-20T03:35:24.080 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.080 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.080 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.081 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298405+0000", 2023-10-20T03:35:24.081 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6470000000000003e-05 2023-10-20T03:35:24.081 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.081 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.082 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.082 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.082 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.082 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=23, last_complete=37'14, committed=1, applied=1))", 2023-10-20T03:35:24.082 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.283349+0000", 2023-10-20T03:35:24.083 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6579314700000003, 2023-10-20T03:35:24.083 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015191886, 2023-10-20T03:35:24.083 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.083 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:24.083 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.083 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.084 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.084 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.283349+0000", 2023-10-20T03:35:24.084 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.084 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.084 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.085 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.085 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.283349+0000", 2023-10-20T03:35:24.085 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.085 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.085 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.086 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.086 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.283351+0000", 2023-10-20T03:35:24.086 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2369999999999999e-06 2023-10-20T03:35:24.086 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.086 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.087 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.087 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.283364+0000", 2023-10-20T03:35:24.087 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2605999999999999e-05 2023-10-20T03:35:24.087 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.087 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.088 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.088 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.283401+0000", 2023-10-20T03:35:24.088 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7154999999999999e-05 2023-10-20T03:35:24.088 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.089 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.089 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.089 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.283422+0000", 2023-10-20T03:35:24.089 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0936000000000001e-05 2023-10-20T03:35:24.089 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.090 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.090 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.090 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298487+0000", 2023-10-20T03:35:24.090 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015064889999999999 2023-10-20T03:35:24.090 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.091 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.091 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.091 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298541+0000", 2023-10-20T03:35:24.092 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4061999999999998e-05 2023-10-20T03:35:24.092 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.092 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.092 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.092 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=23, last_complete=37'14, committed=1, applied=1))", 2023-10-20T03:35:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.289329+0000", 2023-10-20T03:35:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6519509660000002, 2023-10-20T03:35:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0093795019999999996, 2023-10-20T03:35:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.289329+0000", 2023-10-20T03:35:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.289329+0000", 2023-10-20T03:35:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.289331+0000", 2023-10-20T03:35:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.64e-06 2023-10-20T03:35:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.289342+0000", 2023-10-20T03:35:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1076e-05 2023-10-20T03:35:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.289364+0000", 2023-10-20T03:35:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1719e-05 2023-10-20T03:35:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.289377+0000", 2023-10-20T03:35:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3553999999999999e-05 2023-10-20T03:35:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298659+0000", 2023-10-20T03:35:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0092816789999999993 2023-10-20T03:35:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298709+0000", 2023-10-20T03:35:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9833999999999998e-05 2023-10-20T03:35:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=23, last_complete=37'14, committed=1, applied=1))", 2023-10-20T03:35:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.290080+0000", 2023-10-20T03:35:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6512003599999998, 2023-10-20T03:35:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0094360669999999994, 2023-10-20T03:35:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.104 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:24.104 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.104 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.105 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.105 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.290080+0000", 2023-10-20T03:35:24.105 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.105 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.106 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.106 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.106 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.290080+0000", 2023-10-20T03:35:24.106 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.106 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.107 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.107 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.107 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.290082+0000", 2023-10-20T03:35:24.108 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9429999999999999e-06 2023-10-20T03:35:24.108 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.109 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.109 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.109 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.290093+0000", 2023-10-20T03:35:24.110 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0847e-05 2023-10-20T03:35:24.110 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.110 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.111 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.111 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.290125+0000", 2023-10-20T03:35:24.111 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2784999999999999e-05 2023-10-20T03:35:24.112 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.112 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.113 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.113 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.290145+0000", 2023-10-20T03:35:24.113 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9843e-05 2023-10-20T03:35:24.114 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.114 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.114 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.115 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298816+0000", 2023-10-20T03:35:24.115 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.008670278 2023-10-20T03:35:24.116 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.116 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.116 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.116 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299516+0000", 2023-10-20T03:35:24.116 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00070037099999999998 2023-10-20T03:35:24.117 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.117 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.118 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.118 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.118 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.118 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=24, last_complete=37'15, committed=1, applied=1))", 2023-10-20T03:35:24.118 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.294639+0000", 2023-10-20T03:35:24.119 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6466411369999996, 2023-10-20T03:35:24.119 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0050178180000000003, 2023-10-20T03:35:24.119 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.119 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:24.119 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.119 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.120 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.120 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.294639+0000", 2023-10-20T03:35:24.120 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.120 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.120 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.121 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.121 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.294639+0000", 2023-10-20T03:35:24.121 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.121 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.121 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.121 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.122 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.294641+0000", 2023-10-20T03:35:24.122 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.717e-06 2023-10-20T03:35:24.122 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.122 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.122 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.123 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.294651+0000", 2023-10-20T03:35:24.123 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0049e-05 2023-10-20T03:35:24.123 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.123 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.123 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.124 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.294685+0000", 2023-10-20T03:35:24.124 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4625000000000001e-05 2023-10-20T03:35:24.124 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.124 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.124 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.124 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.294708+0000", 2023-10-20T03:35:24.125 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2209e-05 2023-10-20T03:35:24.125 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.125 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.125 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.125 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299598+0000", 2023-10-20T03:35:24.126 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0048907869999999997 2023-10-20T03:35:24.126 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.126 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.126 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.126 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299657+0000", 2023-10-20T03:35:24.126 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8430999999999997e-05 2023-10-20T03:35:24.127 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.127 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.127 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.127 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.127 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.128 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=24, last_complete=37'15, committed=1, applied=1))", 2023-10-20T03:35:24.128 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.294813+0000", 2023-10-20T03:35:24.128 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6464677339999998, 2023-10-20T03:35:24.128 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0050140860000000001, 2023-10-20T03:35:24.128 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.129 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:24.129 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.129 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.129 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.129 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.294813+0000", 2023-10-20T03:35:24.129 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.130 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.130 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.130 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.130 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.294813+0000", 2023-10-20T03:35:24.130 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.131 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.131 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.131 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.294814+0000", 2023-10-20T03:35:24.131 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2869999999999999e-06 2023-10-20T03:35:24.132 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.132 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.132 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.132 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.294821+0000", 2023-10-20T03:35:24.132 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2559999999999996e-06 2023-10-20T03:35:24.132 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.133 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.133 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.133 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.294846+0000", 2023-10-20T03:35:24.133 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5092999999999998e-05 2023-10-20T03:35:24.133 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.134 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.134 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.134 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.294863+0000", 2023-10-20T03:35:24.134 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6633000000000002e-05 2023-10-20T03:35:24.134 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.135 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.135 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.135 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299768+0000", 2023-10-20T03:35:24.135 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0049049389999999997 2023-10-20T03:35:24.135 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.136 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.136 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.136 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299827+0000", 2023-10-20T03:35:24.137 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8878000000000001e-05 2023-10-20T03:35:24.137 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.137 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.137 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.137 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.138 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.138 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=24, last_complete=37'15, committed=1, applied=1))", 2023-10-20T03:35:24.138 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.295027+0000", 2023-10-20T03:35:24.138 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6462530520000005, 2023-10-20T03:35:24.138 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0049206400000000004, 2023-10-20T03:35:24.139 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.139 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:24.139 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.139 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.139 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.139 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.295027+0000", 2023-10-20T03:35:24.140 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.140 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.140 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.140 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.140 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.295027+0000", 2023-10-20T03:35:24.141 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.141 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.141 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.141 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.142 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.295029+0000", 2023-10-20T03:35:24.142 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7099999999999999e-06 2023-10-20T03:35:24.142 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.142 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.142 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.143 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.295040+0000", 2023-10-20T03:35:24.143 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1e-05 2023-10-20T03:35:24.143 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.143 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.143 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.144 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.295075+0000", 2023-10-20T03:35:24.144 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4826000000000002e-05 2023-10-20T03:35:24.144 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.144 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.144 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.145 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.295098+0000", 2023-10-20T03:35:24.145 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3422e-05 2023-10-20T03:35:24.145 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.145 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.146 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.146 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299912+0000", 2023-10-20T03:35:24.146 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0048136999999999997 2023-10-20T03:35:24.146 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.146 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.147 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.147 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299948+0000", 2023-10-20T03:35:24.147 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5982000000000002e-05 2023-10-20T03:35:24.147 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.147 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.148 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.148 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.148 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.148 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=24, last_complete=37'15, committed=1, applied=1))", 2023-10-20T03:35:24.148 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.298344+0000", 2023-10-20T03:35:24.148 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6429363429999997, 2023-10-20T03:35:24.149 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001748845, 2023-10-20T03:35:24.149 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.149 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:24.149 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.149 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.150 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.150 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298344+0000", 2023-10-20T03:35:24.150 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.150 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.150 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.151 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.151 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298344+0000", 2023-10-20T03:35:24.151 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.151 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.151 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.151 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.152 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298346+0000", 2023-10-20T03:35:24.152 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6530000000000001e-06 2023-10-20T03:35:24.152 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.152 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.152 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.153 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298357+0000", 2023-10-20T03:35:24.153 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1343e-05 2023-10-20T03:35:24.153 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.153 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.153 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.154 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298383+0000", 2023-10-20T03:35:24.154 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5715e-05 2023-10-20T03:35:24.154 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.154 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.155 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.155 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298401+0000", 2023-10-20T03:35:24.155 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8272999999999999e-05 2023-10-20T03:35:24.155 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.155 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.156 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.156 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.300041+0000", 2023-10-20T03:35:24.156 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0016400830000000001 2023-10-20T03:35:24.156 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.156 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.157 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.157 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.300093+0000", 2023-10-20T03:35:24.157 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1777999999999998e-05 2023-10-20T03:35:24.157 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.157 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.158 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.158 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.158 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.158 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=24, last_complete=37'15, committed=1, applied=1))", 2023-10-20T03:35:24.158 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.299802+0000", 2023-10-20T03:35:24.159 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.6414782939999997, 2023-10-20T03:35:24.159 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0013190750000000001, 2023-10-20T03:35:24.159 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.159 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:24.159 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.160 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.160 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.160 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299802+0000", 2023-10-20T03:35:24.160 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.160 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.160 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.161 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.161 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299802+0000", 2023-10-20T03:35:24.161 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.161 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.161 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.162 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.162 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299804+0000", 2023-10-20T03:35:24.162 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9039999999999999e-06 2023-10-20T03:35:24.162 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.162 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.163 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.163 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299816+0000", 2023-10-20T03:35:24.163 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2225999999999999e-05 2023-10-20T03:35:24.163 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.163 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.164 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.164 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299840+0000", 2023-10-20T03:35:24.164 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3912000000000001e-05 2023-10-20T03:35:24.164 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.164 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.164 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.165 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299858+0000", 2023-10-20T03:35:24.165 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8223000000000002e-05 2023-10-20T03:35:24.165 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.165 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.165 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.166 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.300175+0000", 2023-10-20T03:35:24.166 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000316831 2023-10-20T03:35:24.166 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.166 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.167 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.167 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.301121+0000", 2023-10-20T03:35:24.167 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000945979 2023-10-20T03:35:24.167 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.167 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.168 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.168 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.168 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.7s1 38/31 ECSubWrite(tid=63, reqid=client.4570.0:216, at_version=38'7, trim_to=0'0, roll_forward_to=34'5))", 2023-10-20T03:35:24.168 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:20.273399+0000", 2023-10-20T03:35:24.169 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 3.667881393, 2023-10-20T03:35:24.169 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.003194672, 2023-10-20T03:35:24.169 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.169 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:24.169 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.170 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.170 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.170 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.273399+0000", 2023-10-20T03:35:24.170 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.170 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.171 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.171 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.171 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.273399+0000", 2023-10-20T03:35:24.171 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.171 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.171 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.172 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.172 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.273399+0000", 2023-10-20T03:35:24.172 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3000000000000001e-07 2023-10-20T03:35:24.172 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.172 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.173 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.173 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.273446+0000", 2023-10-20T03:35:24.173 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6418000000000002e-05 2023-10-20T03:35:24.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.173 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.173 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.174 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.273470+0000", 2023-10-20T03:35:24.174 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3935999999999999e-05 2023-10-20T03:35:24.174 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.174 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.174 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.175 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.273486+0000", 2023-10-20T03:35:24.175 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6116e-05 2023-10-20T03:35:24.175 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.175 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.175 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.176 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.273674+0000", 2023-10-20T03:35:24.176 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000187997 2023-10-20T03:35:24.176 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.176 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.176 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:24.177 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.273706+0000", 2023-10-20T03:35:24.177 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2582000000000001e-05 2023-10-20T03:35:24.177 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.177 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.177 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:24.177 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.276518+0000", 2023-10-20T03:35:24.178 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0028111899999999999 2023-10-20T03:35:24.178 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.178 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.178 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.178 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.276594+0000", 2023-10-20T03:35:24.179 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6002999999999997e-05 2023-10-20T03:35:24.179 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.179 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.179 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.179 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.180 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.180 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.7s1 38/31 ECSubWrite(tid=64, reqid=client.4570.0:216, at_version=0'0, trim_to=0'0, roll_forward_to=38'7))", 2023-10-20T03:35:24.180 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:20.279200+0000", 2023-10-20T03:35:24.180 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 3.6620797870000001, 2023-10-20T03:35:24.180 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0014819200000000001, 2023-10-20T03:35:24.180 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.181 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:24.181 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.181 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.181 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.181 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.279200+0000", 2023-10-20T03:35:24.182 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.182 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.182 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.182 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.182 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.279200+0000", 2023-10-20T03:35:24.183 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.183 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.183 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.183 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.183 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.279202+0000", 2023-10-20T03:35:24.184 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8840000000000001e-06 2023-10-20T03:35:24.184 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.184 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.184 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.184 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.279238+0000", 2023-10-20T03:35:24.185 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5658999999999999e-05 2023-10-20T03:35:24.185 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.185 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.185 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.186 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.279287+0000", 2023-10-20T03:35:24.186 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9070999999999998e-05 2023-10-20T03:35:24.186 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.186 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.186 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.187 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.279314+0000", 2023-10-20T03:35:24.187 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6752000000000001e-05 2023-10-20T03:35:24.187 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.187 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.187 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.188 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.279416+0000", 2023-10-20T03:35:24.188 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000102575 2023-10-20T03:35:24.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.188 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.188 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:24.189 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.279429+0000", 2023-10-20T03:35:24.189 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2967e-05 2023-10-20T03:35:24.189 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.189 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.189 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:24.190 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.280607+0000", 2023-10-20T03:35:24.190 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0011773879999999999 2023-10-20T03:35:24.190 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.190 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.190 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.191 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:20.280682+0000", 2023-10-20T03:35:24.191 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5624000000000006e-05 2023-10-20T03:35:24.191 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.191 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.191 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.192 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.192 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.192 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:24.194 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-10-20T03:35:24.194 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:24.190+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:24.294 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:24.290+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:24.304 INFO:tasks.rados.rados.0.smithi170.stdout:115: done (4 left) 2023-10-20T03:35:24.304 INFO:tasks.rados.rados.0.smithi170.stdout:116: done (3 left) 2023-10-20T03:35:24.305 INFO:tasks.rados.rados.0.smithi170.stdout:117: done (2 left) 2023-10-20T03:35:24.305 INFO:tasks.rados.rados.0.smithi170.stdout:118: done (1 left) 2023-10-20T03:35:24.305 INFO:tasks.rados.rados.0.smithi170.stdout:119: done (0 left) 2023-10-20T03:35:24.305 INFO:tasks.rados.rados.0.smithi170.stdout:120: snap_create 2023-10-20T03:35:24.364 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:24.365 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:24.378 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-10-20T03:35:24.395 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:24.393+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:24.440 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:24.441 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:35:24.446 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:24.447 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:24.496 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:24.493+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:24.542 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:24.543 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:24.543 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:35:24.543 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:35:24.543 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:24.557 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-10-20T03:35:24.597 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:24.596+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:24.682 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:24.683 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2023-10-20T03:35:24.683 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2023-10-20T03:35:24.683 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2023-10-20T03:35:24.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.683 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:118 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'80, mlcod=24'78)", 2023-10-20T03:35:24.684 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:02.681656+0000", 2023-10-20T03:35:24.684 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.998789734999999, 2023-10-20T03:35:24.684 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0039841369999999996, 2023-10-20T03:35:24.684 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.684 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.685 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.685 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.685 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.681656+0000", 2023-10-20T03:35:24.685 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.686 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.686 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.686 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.681656+0000", 2023-10-20T03:35:24.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.681657+0000", 2023-10-20T03:35:24.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6270000000000001e-06 2023-10-20T03:35:24.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.681752+0000", 2023-10-20T03:35:24.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4678000000000004e-05 2023-10-20T03:35:24.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.689 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.689 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.681778+0000", 2023-10-20T03:35:24.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6137999999999999e-05 2023-10-20T03:35:24.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.690 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.690 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.681791+0000", 2023-10-20T03:35:24.690 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2839e-05 2023-10-20T03:35:24.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.691 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.691 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.681876+0000", 2023-10-20T03:35:24.691 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5116000000000003e-05 2023-10-20T03:35:24.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.692 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.681910+0000", 2023-10-20T03:35:24.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3847999999999998e-05 2023-10-20T03:35:24.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.693 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.685568+0000", 2023-10-20T03:35:24.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0036584590000000002 2023-10-20T03:35:24.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.685640+0000", 2023-10-20T03:35:24.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.1432000000000003e-05 2023-10-20T03:35:24.694 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.694 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.695 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.695 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.695 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.695 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:120 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'82, mlcod=24'78)", 2023-10-20T03:35:24.695 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:02.685865+0000", 2023-10-20T03:35:24.696 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.994580537000001, 2023-10-20T03:35:24.696 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0045888279999999997, 2023-10-20T03:35:24.696 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.696 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.696 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.685865+0000", 2023-10-20T03:35:24.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.685865+0000", 2023-10-20T03:35:24.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.699 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.685866+0000", 2023-10-20T03:35:24.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3829999999999999e-06 2023-10-20T03:35:24.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.699 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.699 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.685949+0000", 2023-10-20T03:35:24.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2842000000000004e-05 2023-10-20T03:35:24.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.700 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.685977+0000", 2023-10-20T03:35:24.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7325e-05 2023-10-20T03:35:24.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.685989+0000", 2023-10-20T03:35:24.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2806000000000001e-05 2023-10-20T03:35:24.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.702 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.702 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.686091+0000", 2023-10-20T03:35:24.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101881 2023-10-20T03:35:24.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.686168+0000", 2023-10-20T03:35:24.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7265e-05 2023-10-20T03:35:24.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.704 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.704 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.690337+0000", 2023-10-20T03:35:24.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0041688960000000001 2023-10-20T03:35:24.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.690454+0000", 2023-10-20T03:35:24.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011642999999999999 2023-10-20T03:35:24.705 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.705 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.706 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.706 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:143 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'101, mlcod=24'98)", 2023-10-20T03:35:24.706 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:02.884473+0000", 2023-10-20T03:35:24.707 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.795972943999999, 2023-10-20T03:35:24.707 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0044148950000000003, 2023-10-20T03:35:24.707 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.707 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.707 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.707 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.708 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.708 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.884473+0000", 2023-10-20T03:35:24.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.708 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.884473+0000", 2023-10-20T03:35:24.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.884474+0000", 2023-10-20T03:35:24.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.226e-06 2023-10-20T03:35:24.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.884526+0000", 2023-10-20T03:35:24.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2253999999999998e-05 2023-10-20T03:35:24.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.884546+0000", 2023-10-20T03:35:24.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9616e-05 2023-10-20T03:35:24.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.884558+0000", 2023-10-20T03:35:24.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2568999999999999e-05 2023-10-20T03:35:24.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.713 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.713 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.884699+0000", 2023-10-20T03:35:24.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014053199999999999 2023-10-20T03:35:24.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.884736+0000", 2023-10-20T03:35:24.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6773999999999997e-05 2023-10-20T03:35:24.715 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.715 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.715 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.715 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.888807+0000", 2023-10-20T03:35:24.715 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0040713449999999997 2023-10-20T03:35:24.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.716 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.716 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.716 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.888887+0000", 2023-10-20T03:35:24.716 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0579000000000002e-05 2023-10-20T03:35:24.717 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.717 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.717 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.717 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:144 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'102, mlcod=24'98)", 2023-10-20T03:35:24.718 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:02.886772+0000", 2023-10-20T03:35:24.718 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.793673757000001, 2023-10-20T03:35:24.718 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0042510810000000003, 2023-10-20T03:35:24.718 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.719 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.719 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.719 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.719 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.886772+0000", 2023-10-20T03:35:24.719 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.886772+0000", 2023-10-20T03:35:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.886773+0000", 2023-10-20T03:35:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.66e-06 2023-10-20T03:35:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.886867+0000", 2023-10-20T03:35:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3097999999999995e-05 2023-10-20T03:35:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.886897+0000", 2023-10-20T03:35:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0120999999999998e-05 2023-10-20T03:35:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.886915+0000", 2023-10-20T03:35:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8675999999999999e-05 2023-10-20T03:35:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.887066+0000", 2023-10-20T03:35:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015083800000000001 2023-10-20T03:35:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.887119+0000", 2023-10-20T03:35:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2927000000000003e-05 2023-10-20T03:35:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.890942+0000", 2023-10-20T03:35:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0038232729999999999 2023-10-20T03:35:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:02.891023+0000", 2023-10-20T03:35:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0488000000000006e-05 2023-10-20T03:35:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:153 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'107, mlcod=24'105)", 2023-10-20T03:35:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.050059+0000", 2023-10-20T03:35:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.630386546, 2023-10-20T03:35:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0049452029999999996, 2023-10-20T03:35:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050059+0000", 2023-10-20T03:35:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050059+0000", 2023-10-20T03:35:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050061+0000", 2023-10-20T03:35:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1229999999999998e-06 2023-10-20T03:35:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050077+0000", 2023-10-20T03:35:24.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5866e-05 2023-10-20T03:35:24.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.734 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.734 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050112+0000", 2023-10-20T03:35:24.735 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5274000000000001e-05 2023-10-20T03:35:24.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.735 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.735 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050131+0000", 2023-10-20T03:35:24.736 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8946000000000001e-05 2023-10-20T03:35:24.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.736 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.736 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.736 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050244+0000", 2023-10-20T03:35:24.736 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011243699999999999 2023-10-20T03:35:24.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.737 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.737 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.737 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050302+0000", 2023-10-20T03:35:24.737 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8396000000000001e-05 2023-10-20T03:35:24.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.738 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.738 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.738 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054943+0000", 2023-10-20T03:35:24.738 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0046406909999999997 2023-10-20T03:35:24.738 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.739 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.739 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.739 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055004+0000", 2023-10-20T03:35:24.739 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1470000000000006e-05 2023-10-20T03:35:24.739 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.739 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.740 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.740 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.740 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:154 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'108, mlcod=24'105)", 2023-10-20T03:35:24.740 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.050423+0000", 2023-10-20T03:35:24.740 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.630022063999998, 2023-10-20T03:35:24.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.004629518, 2023-10-20T03:35:24.741 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.741 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.741 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.741 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.742 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050423+0000", 2023-10-20T03:35:24.742 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.743 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050423+0000", 2023-10-20T03:35:24.743 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.743 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.743 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.743 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.744 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050425+0000", 2023-10-20T03:35:24.744 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6169999999999999e-06 2023-10-20T03:35:24.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.744 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.744 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.744 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050439+0000", 2023-10-20T03:35:24.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3516000000000001e-05 2023-10-20T03:35:24.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.745 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.745 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050463+0000", 2023-10-20T03:35:24.746 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4905e-05 2023-10-20T03:35:24.746 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.746 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.746 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.050483+0000", 2023-10-20T03:35:24.747 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9592000000000001e-05 2023-10-20T03:35:24.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.747 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.747 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.747 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051173+0000", 2023-10-20T03:35:24.747 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00068973799999999996 2023-10-20T03:35:24.748 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.748 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.748 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.748 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051226+0000", 2023-10-20T03:35:24.748 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3393999999999997e-05 2023-10-20T03:35:24.748 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.749 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.749 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.749 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055013+0000", 2023-10-20T03:35:24.749 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0037868680000000001 2023-10-20T03:35:24.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.750 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.750 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.750 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055053+0000", 2023-10-20T03:35:24.750 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9888000000000002e-05 2023-10-20T03:35:24.750 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.751 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.751 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.751 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.751 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.751 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:157 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'111, mlcod=24'105)", 2023-10-20T03:35:24.751 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.051811+0000", 2023-10-20T03:35:24.752 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.628634796, 2023-10-20T03:35:24.752 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0082276899999999993, 2023-10-20T03:35:24.752 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.752 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.752 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.753 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.753 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051811+0000", 2023-10-20T03:35:24.753 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.753 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.753 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.754 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.754 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051811+0000", 2023-10-20T03:35:24.754 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.754 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.755 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.755 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.755 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051813+0000", 2023-10-20T03:35:24.755 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0470000000000001e-06 2023-10-20T03:35:24.755 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.755 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.756 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.756 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051821+0000", 2023-10-20T03:35:24.756 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1759999999999993e-06 2023-10-20T03:35:24.756 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.756 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.757 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.757 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051834+0000", 2023-10-20T03:35:24.757 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3219e-05 2023-10-20T03:35:24.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.757 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.757 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.758 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051844+0000", 2023-10-20T03:35:24.758 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4569999999999994e-06 2023-10-20T03:35:24.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.758 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.759 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053007+0000", 2023-10-20T03:35:24.759 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00116373 2023-10-20T03:35:24.759 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.759 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.759 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.760 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053036+0000", 2023-10-20T03:35:24.760 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8615e-05 2023-10-20T03:35:24.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.760 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.760 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.760 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.059974+0000", 2023-10-20T03:35:24.761 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0069383539999999999 2023-10-20T03:35:24.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.761 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.761 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.761 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060038+0000", 2023-10-20T03:35:24.762 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4091999999999996e-05 2023-10-20T03:35:24.762 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.762 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.762 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.763 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.763 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:158 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'112, mlcod=24'105)", 2023-10-20T03:35:24.763 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.052405+0000", 2023-10-20T03:35:24.763 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.628040296000002, 2023-10-20T03:35:24.763 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0076727749999999997, 2023-10-20T03:35:24.763 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.764 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.764 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.764 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.764 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.764 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052405+0000", 2023-10-20T03:35:24.765 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.765 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.765 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.765 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.765 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052405+0000", 2023-10-20T03:35:24.766 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.766 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.766 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.766 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.766 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052406+0000", 2023-10-20T03:35:24.767 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.237e-06 2023-10-20T03:35:24.767 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.767 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.767 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.767 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052462+0000", 2023-10-20T03:35:24.768 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5203e-05 2023-10-20T03:35:24.768 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.768 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.768 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.768 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052477+0000", 2023-10-20T03:35:24.768 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5559000000000001e-05 2023-10-20T03:35:24.769 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.769 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.769 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.769 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052488+0000", 2023-10-20T03:35:24.769 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0949e-05 2023-10-20T03:35:24.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.770 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.770 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.770 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053461+0000", 2023-10-20T03:35:24.770 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00097277500000000005 2023-10-20T03:35:24.771 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.771 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.771 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.771 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053488+0000", 2023-10-20T03:35:24.771 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7322e-05 2023-10-20T03:35:24.771 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060041+0000", 2023-10-20T03:35:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0065531290000000004 2023-10-20T03:35:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.773 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.773 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.773 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060078+0000", 2023-10-20T03:35:24.773 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6600999999999997e-05 2023-10-20T03:35:24.773 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.774 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.774 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.774 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.774 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.774 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:159 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'113, mlcod=24'105)", 2023-10-20T03:35:24.775 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.052999+0000", 2023-10-20T03:35:24.775 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.627446890000002, 2023-10-20T03:35:24.775 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0071362159999999999, 2023-10-20T03:35:24.775 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.775 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.775 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.776 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.776 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.776 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052999+0000", 2023-10-20T03:35:24.776 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.777 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.777 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052999+0000", 2023-10-20T03:35:24.777 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.778 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.778 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053000+0000", 2023-10-20T03:35:24.778 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5969999999999999e-06 2023-10-20T03:35:24.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.778 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.779 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.779 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053012+0000", 2023-10-20T03:35:24.779 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1299e-05 2023-10-20T03:35:24.779 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.779 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.779 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.780 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053035+0000", 2023-10-20T03:35:24.780 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3451999999999998e-05 2023-10-20T03:35:24.780 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.780 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.780 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.781 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053048+0000", 2023-10-20T03:35:24.781 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2563e-05 2023-10-20T03:35:24.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.781 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.781 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.781 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053976+0000", 2023-10-20T03:35:24.782 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000928764 2023-10-20T03:35:24.782 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.782 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.782 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.782 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054004+0000", 2023-10-20T03:35:24.783 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7957999999999999e-05 2023-10-20T03:35:24.783 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.783 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.783 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.783 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060080+0000", 2023-10-20T03:35:24.784 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0060754789999999999 2023-10-20T03:35:24.784 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.784 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.784 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.784 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060135+0000", 2023-10-20T03:35:24.784 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5104e-05 2023-10-20T03:35:24.785 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.785 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.785 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.785 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.785 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.786 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:160 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'114, mlcod=24'105)", 2023-10-20T03:35:24.786 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.053502+0000", 2023-10-20T03:35:24.786 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.626943877999999, 2023-10-20T03:35:24.786 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.006700157, 2023-10-20T03:35:24.786 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.787 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.787 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.787 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.787 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.787 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053502+0000", 2023-10-20T03:35:24.788 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.788 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.788 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.788 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.788 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053502+0000", 2023-10-20T03:35:24.788 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.789 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.789 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.789 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.789 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053503+0000", 2023-10-20T03:35:24.789 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0899999999999999e-06 2023-10-20T03:35:24.790 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.790 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.790 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.790 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053512+0000", 2023-10-20T03:35:24.790 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6290000000000005e-06 2023-10-20T03:35:24.790 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.791 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.791 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.791 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053528+0000", 2023-10-20T03:35:24.791 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5156e-05 2023-10-20T03:35:24.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.792 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.792 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.792 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053537+0000", 2023-10-20T03:35:24.792 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0359999999999995e-06 2023-10-20T03:35:24.792 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.793 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.793 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.793 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054446+0000", 2023-10-20T03:35:24.793 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000909105 2023-10-20T03:35:24.793 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.794 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.794 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.794 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054479+0000", 2023-10-20T03:35:24.794 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3754999999999998e-05 2023-10-20T03:35:24.794 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.795 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.795 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.795 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060138+0000", 2023-10-20T03:35:24.795 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0056582220000000001 2023-10-20T03:35:24.795 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.796 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.796 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.796 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060202+0000", 2023-10-20T03:35:24.796 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4164000000000004e-05 2023-10-20T03:35:24.796 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.796 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.797 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.797 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.797 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.797 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:161 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'115, mlcod=24'105)", 2023-10-20T03:35:24.797 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.054101+0000", 2023-10-20T03:35:24.798 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.626344275000001, 2023-10-20T03:35:24.798 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.006183132, 2023-10-20T03:35:24.798 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.798 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.798 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.798 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.799 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.799 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054101+0000", 2023-10-20T03:35:24.799 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.799 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.799 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.800 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.800 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054101+0000", 2023-10-20T03:35:24.800 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.800 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.800 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.801 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.801 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054103+0000", 2023-10-20T03:35:24.801 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9360000000000002e-06 2023-10-20T03:35:24.801 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.801 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.801 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.802 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054170+0000", 2023-10-20T03:35:24.802 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6422999999999997e-05 2023-10-20T03:35:24.802 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.802 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.802 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.803 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054194+0000", 2023-10-20T03:35:24.803 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4629000000000001e-05 2023-10-20T03:35:24.803 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.803 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.803 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.804 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054209+0000", 2023-10-20T03:35:24.804 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5271999999999999e-05 2023-10-20T03:35:24.804 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.804 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.804 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.804 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055209+0000", 2023-10-20T03:35:24.805 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00099963000000000005 2023-10-20T03:35:24.805 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.805 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.805 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.805 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055239+0000", 2023-10-20T03:35:24.806 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9505e-05 2023-10-20T03:35:24.806 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.806 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.806 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.806 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060208+0000", 2023-10-20T03:35:24.807 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0049690649999999999 2023-10-20T03:35:24.807 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.807 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.807 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.807 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060284+0000", 2023-10-20T03:35:24.807 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6672e-05 2023-10-20T03:35:24.808 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.808 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.808 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.808 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.808 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.809 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:162 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'116, mlcod=24'105)", 2023-10-20T03:35:24.809 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.054961+0000", 2023-10-20T03:35:24.809 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.625484264000001, 2023-10-20T03:35:24.809 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0053669989999999999, 2023-10-20T03:35:24.809 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.809 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.810 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.810 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.810 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.810 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054961+0000", 2023-10-20T03:35:24.810 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.811 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.811 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.811 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.811 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054961+0000", 2023-10-20T03:35:24.811 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.811 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.812 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.812 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.812 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054963+0000", 2023-10-20T03:35:24.812 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9700000000000002e-06 2023-10-20T03:35:24.812 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.812 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.813 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.813 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.054978+0000", 2023-10-20T03:35:24.813 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5115e-05 2023-10-20T03:35:24.813 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.813 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.814 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.814 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055008+0000", 2023-10-20T03:35:24.814 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0076000000000001e-05 2023-10-20T03:35:24.814 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.814 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.815 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.815 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055025+0000", 2023-10-20T03:35:24.815 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6694999999999999e-05 2023-10-20T03:35:24.815 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.815 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.816 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.816 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.056990+0000", 2023-10-20T03:35:24.816 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0019646149999999999 2023-10-20T03:35:24.816 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.816 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.817 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.817 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.057044+0000", 2023-10-20T03:35:24.817 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3946999999999997e-05 2023-10-20T03:35:24.817 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.817 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.817 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060287+0000", 2023-10-20T03:35:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.003242923 2023-10-20T03:35:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060328+0000", 2023-10-20T03:35:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1658e-05 2023-10-20T03:35:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:163 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'117, mlcod=24'105)", 2023-10-20T03:35:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.055379+0000", 2023-10-20T03:35:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.625066060999998, 2023-10-20T03:35:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0051660530000000003, 2023-10-20T03:35:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055379+0000", 2023-10-20T03:35:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055379+0000", 2023-10-20T03:35:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055380+0000", 2023-10-20T03:35:24.824 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.04e-06 2023-10-20T03:35:24.824 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.824 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.824 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.824 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055389+0000", 2023-10-20T03:35:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4659999999999999e-06 2023-10-20T03:35:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055416+0000", 2023-10-20T03:35:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7498e-05 2023-10-20T03:35:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055428+0000", 2023-10-20T03:35:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.172e-05 2023-10-20T03:35:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.057647+0000", 2023-10-20T03:35:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002219127 2023-10-20T03:35:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.057679+0000", 2023-10-20T03:35:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2122000000000002e-05 2023-10-20T03:35:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060477+0000", 2023-10-20T03:35:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0027973910000000002 2023-10-20T03:35:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060545+0000", 2023-10-20T03:35:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8689000000000006e-05 2023-10-20T03:35:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.831 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.831 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.831 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.831 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:164 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'118, mlcod=24'105)", 2023-10-20T03:35:24.831 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.055917+0000", 2023-10-20T03:35:24.832 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.624528732000002, 2023-10-20T03:35:24.832 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0057593599999999998, 2023-10-20T03:35:24.832 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.832 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.832 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.832 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.833 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.833 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055917+0000", 2023-10-20T03:35:24.833 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.833 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.833 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.834 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.834 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055917+0000", 2023-10-20T03:35:24.834 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.834 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.835 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.835 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.055918+0000", 2023-10-20T03:35:24.835 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4869999999999999e-06 2023-10-20T03:35:24.835 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.835 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.835 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.836 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.056063+0000", 2023-10-20T03:35:24.836 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000144709 2023-10-20T03:35:24.836 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.836 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.836 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.837 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.056085+0000", 2023-10-20T03:35:24.837 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1827999999999998e-05 2023-10-20T03:35:24.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.837 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.837 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.838 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.056096+0000", 2023-10-20T03:35:24.838 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1116e-05 2023-10-20T03:35:24.838 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.838 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.838 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.839 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.058154+0000", 2023-10-20T03:35:24.839 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020580170000000001 2023-10-20T03:35:24.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.839 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.839 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.839 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.058181+0000", 2023-10-20T03:35:24.840 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7412000000000001e-05 2023-10-20T03:35:24.840 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.840 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.840 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.840 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.061621+0000", 2023-10-20T03:35:24.840 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0034400640000000001 2023-10-20T03:35:24.841 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.841 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.841 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.841 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.061676+0000", 2023-10-20T03:35:24.841 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4727e-05 2023-10-20T03:35:24.842 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.842 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.842 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.842 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.842 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.843 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:165 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'119, mlcod=25'109)", 2023-10-20T03:35:24.843 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.057404+0000", 2023-10-20T03:35:24.843 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.623041989000001, 2023-10-20T03:35:24.843 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0043555130000000001, 2023-10-20T03:35:24.843 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.844 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.844 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.844 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.844 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.844 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.057404+0000", 2023-10-20T03:35:24.844 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.845 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.845 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.845 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.845 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.057404+0000", 2023-10-20T03:35:24.845 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.846 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.846 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.846 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.846 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.057405+0000", 2023-10-20T03:35:24.846 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1829999999999999e-06 2023-10-20T03:35:24.847 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.847 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.847 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.847 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.057416+0000", 2023-10-20T03:35:24.847 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1195999999999999e-05 2023-10-20T03:35:24.847 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.848 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.848 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.848 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.057433+0000", 2023-10-20T03:35:24.848 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6991999999999999e-05 2023-10-20T03:35:24.848 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.057444+0000", 2023-10-20T03:35:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0993e-05 2023-10-20T03:35:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.059452+0000", 2023-10-20T03:35:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020077860000000001 2023-10-20T03:35:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.059485+0000", 2023-10-20T03:35:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3701000000000001e-05 2023-10-20T03:35:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.061681+0000", 2023-10-20T03:35:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002195649 2023-10-20T03:35:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.061759+0000", 2023-10-20T03:35:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8013e-05 2023-10-20T03:35:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:167 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'121, mlcod=25'118)", 2023-10-20T03:35:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.064560+0000", 2023-10-20T03:35:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.615885542000001, 2023-10-20T03:35:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0055001410000000001, 2023-10-20T03:35:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.064560+0000", 2023-10-20T03:35:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.064560+0000", 2023-10-20T03:35:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.064562+0000", 2023-10-20T03:35:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7439999999999999e-06 2023-10-20T03:35:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.064613+0000", 2023-10-20T03:35:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1607000000000002e-05 2023-10-20T03:35:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.064643+0000", 2023-10-20T03:35:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9348000000000001e-05 2023-10-20T03:35:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.862 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.862 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.064656+0000", 2023-10-20T03:35:24.862 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2943e-05 2023-10-20T03:35:24.862 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.862 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.863 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.863 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.065261+0000", 2023-10-20T03:35:24.863 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00060549200000000001 2023-10-20T03:35:24.863 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.863 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.864 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.864 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.065299+0000", 2023-10-20T03:35:24.864 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7997999999999998e-05 2023-10-20T03:35:24.864 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.864 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.864 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.865 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.069991+0000", 2023-10-20T03:35:24.865 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0046922730000000003 2023-10-20T03:35:24.865 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.865 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.865 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.866 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.070060+0000", 2023-10-20T03:35:24.866 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8736000000000001e-05 2023-10-20T03:35:24.866 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.866 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.866 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.867 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.867 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.867 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:168 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'122, mlcod=25'118)", 2023-10-20T03:35:24.867 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.065523+0000", 2023-10-20T03:35:24.867 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.614922629999999, 2023-10-20T03:35:24.868 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0046093719999999996, 2023-10-20T03:35:24.868 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.868 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.868 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.868 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.868 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.869 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.065523+0000", 2023-10-20T03:35:24.869 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.869 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.869 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.869 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.870 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.065523+0000", 2023-10-20T03:35:24.870 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.870 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.870 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.871 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.871 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.065525+0000", 2023-10-20T03:35:24.871 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8870000000000001e-06 2023-10-20T03:35:24.871 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.871 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.872 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.872 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.065592+0000", 2023-10-20T03:35:24.872 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.7065999999999998e-05 2023-10-20T03:35:24.872 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.872 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.873 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.873 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.065619+0000", 2023-10-20T03:35:24.873 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7668999999999999e-05 2023-10-20T03:35:24.873 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.873 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.874 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.874 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.065636+0000", 2023-10-20T03:35:24.874 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6801999999999999e-05 2023-10-20T03:35:24.874 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.874 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.875 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.875 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.066867+0000", 2023-10-20T03:35:24.875 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00123032 2023-10-20T03:35:24.875 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.875 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.875 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.876 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.066897+0000", 2023-10-20T03:35:24.876 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0448000000000001e-05 2023-10-20T03:35:24.876 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.876 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.876 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.877 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.070065+0000", 2023-10-20T03:35:24.877 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0031675969999999999 2023-10-20T03:35:24.877 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.877 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.877 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.878 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.070132+0000", 2023-10-20T03:35:24.878 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.7583000000000007e-05 2023-10-20T03:35:24.878 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.878 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.878 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.879 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.879 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.879 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:169 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'123, mlcod=25'118)", 2023-10-20T03:35:24.879 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.066739+0000", 2023-10-20T03:35:24.879 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.613706089000001, 2023-10-20T03:35:24.880 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0041842060000000002, 2023-10-20T03:35:24.880 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.880 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.880 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.880 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.881 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.881 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.066739+0000", 2023-10-20T03:35:24.881 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.881 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.881 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.882 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.882 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.066739+0000", 2023-10-20T03:35:24.882 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.882 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.882 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.883 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.883 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.066741+0000", 2023-10-20T03:35:24.883 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.753e-06 2023-10-20T03:35:24.883 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.883 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.884 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.884 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.066810+0000", 2023-10-20T03:35:24.884 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8416000000000004e-05 2023-10-20T03:35:24.884 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.884 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.885 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.885 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.066835+0000", 2023-10-20T03:35:24.885 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5058999999999998e-05 2023-10-20T03:35:24.885 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.885 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.885 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.886 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.066850+0000", 2023-10-20T03:35:24.886 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5719e-05 2023-10-20T03:35:24.886 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.886 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.886 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.887 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.068700+0000", 2023-10-20T03:35:24.887 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0018494150000000001 2023-10-20T03:35:24.887 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.887 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.887 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.888 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.068739+0000", 2023-10-20T03:35:24.888 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8751999999999998e-05 2023-10-20T03:35:24.888 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.888 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.888 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.889 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.070858+0000", 2023-10-20T03:35:24.889 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0021194949999999999 2023-10-20T03:35:24.889 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.889 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.070924+0000", 2023-10-20T03:35:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5597000000000002e-05 2023-10-20T03:35:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:203 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'149, mlcod=25'145)", 2023-10-20T03:35:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:05.076293+0000", 2023-10-20T03:35:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 19.604152535000001, 2023-10-20T03:35:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0049082199999999996, 2023-10-20T03:35:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.076293+0000", 2023-10-20T03:35:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.076293+0000", 2023-10-20T03:35:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.076295+0000", 2023-10-20T03:35:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3029999999999998e-06 2023-10-20T03:35:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.076470+0000", 2023-10-20T03:35:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017490499999999999 2023-10-20T03:35:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.076500+0000", 2023-10-20T03:35:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9461000000000001e-05 2023-10-20T03:35:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.076525+0000", 2023-10-20T03:35:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5446e-05 2023-10-20T03:35:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.076874+0000", 2023-10-20T03:35:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00034864100000000001 2023-10-20T03:35:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.076903+0000", 2023-10-20T03:35:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9204999999999999e-05 2023-10-20T03:35:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.081135+0000", 2023-10-20T03:35:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.004231589 2023-10-20T03:35:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.081201+0000", 2023-10-20T03:35:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6669999999999997e-05 2023-10-20T03:35:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:204 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'150, mlcod=25'145)", 2023-10-20T03:35:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:05.076984+0000", 2023-10-20T03:35:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 19.603461862, 2023-10-20T03:35:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0042613950000000003, 2023-10-20T03:35:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.076984+0000", 2023-10-20T03:35:24.904 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.904 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.904 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.904 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:24.904 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.076984+0000", 2023-10-20T03:35:24.905 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:24.905 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.905 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.905 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:24.905 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.076985+0000", 2023-10-20T03:35:24.906 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7969999999999999e-06 2023-10-20T03:35:24.906 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.906 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.906 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:24.906 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.077008+0000", 2023-10-20T03:35:24.907 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2795000000000001e-05 2023-10-20T03:35:24.907 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.907 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.907 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:24.907 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.077034+0000", 2023-10-20T03:35:24.908 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5646000000000001e-05 2023-10-20T03:35:24.908 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.908 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.908 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:24.908 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.077061+0000", 2023-10-20T03:35:24.908 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7300999999999998e-05 2023-10-20T03:35:24.909 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.909 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.909 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:24.909 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.077779+0000", 2023-10-20T03:35:24.909 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00071750500000000001 2023-10-20T03:35:24.909 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.910 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.910 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:24.910 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.077808+0000", 2023-10-20T03:35:24.910 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9122e-05 2023-10-20T03:35:24.910 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.911 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.911 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:24.911 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.081204+0000", 2023-10-20T03:35:24.911 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033964780000000001 2023-10-20T03:35:24.911 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:24.911 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:24.912 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:24.912 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:05.081245+0000", 2023-10-20T03:35:24.912 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0751000000000003e-05 2023-10-20T03:35:24.912 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.912 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.913 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.913 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:24.913 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:24.913 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:24.915 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-10-20T03:35:24.915 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:24.913+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:25.015 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:25.013+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:25.050 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:25.051 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:25.051 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:25.051 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:25.065 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-10-20T03:35:25.150 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:25.149+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:25.218 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:25.214+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:25.235 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:25.235 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:25.236 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:35:25.236 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:35:25.236 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:25.250 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-10-20T03:35:25.307 INFO:tasks.rados.rados.0.smithi170.stdout:120: done (0 left) 2023-10-20T03:35:25.307 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 31 current snap is 7 2023-10-20T03:35:25.307 INFO:tasks.rados.rados.0.smithi170.stdout:121: seq_num 78 ranges {507904=557056,1064960=671744,1736704=98304} 2023-10-20T03:35:25.319 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:25.318+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:25.324 INFO:tasks.rados.rados.0.smithi170.stdout:121: writing smithi17039300-31 from 507904 to 1064960 tid 1 2023-10-20T03:35:25.325 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-10-20T03:35:25.336 INFO:tasks.rados.rados.0.smithi170.stdout:121: writing smithi17039300-31 from 1064960 to 1736704 tid 2 2023-10-20T03:35:25.338 INFO:tasks.rados.rados.0.smithi170.stdout:121: writing smithi17039300-31 from 1736704 to 1835008 tid 3 2023-10-20T03:35:25.338 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 14 current snap is 7 2023-10-20T03:35:25.338 INFO:tasks.rados.rados.0.smithi170.stdout:122: seq_num 79 ranges {229376=573440,802816=425984,1228800=344064} 2023-10-20T03:35:25.351 INFO:tasks.rados.rados.0.smithi170.stdout:122: writing smithi17039300-14 from 229376 to 802816 tid 1 2023-10-20T03:35:25.359 INFO:tasks.rados.rados.0.smithi170.stdout:122: writing smithi17039300-14 from 802816 to 1228800 tid 2 2023-10-20T03:35:25.364 INFO:tasks.rados.rados.0.smithi170.stdout:122: writing smithi17039300-14 from 1228800 to 1572864 tid 3 2023-10-20T03:35:25.364 INFO:tasks.rados.rados.0.smithi170.stdout:123: rollback oid 29 current snap is 7 2023-10-20T03:35:25.364 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 29 to 5 2023-10-20T03:35:25.364 INFO:tasks.rados.rados.0.smithi170.stdout:124: rmattr oid 48 current snap is 7 2023-10-20T03:35:25.365 INFO:tasks.rados.rados.0.smithi170.stdout:124: done (3 left) 2023-10-20T03:35:25.365 INFO:tasks.rados.rados.0.smithi170.stdout:125: snap_create 2023-10-20T03:35:25.366 INFO:tasks.rados.rados.0.smithi170.stdout:121: finishing write tid 1 to smithi17039300-31 2023-10-20T03:35:25.366 INFO:tasks.rados.rados.0.smithi170.stdout:121: finishing write tid 2 to smithi17039300-31 2023-10-20T03:35:25.374 INFO:tasks.rados.rados.0.smithi170.stdout:121: finishing write tid 3 to smithi17039300-31 2023-10-20T03:35:25.374 INFO:tasks.rados.rados.0.smithi170.stdout:121: finishing write tid 4 to smithi17039300-31 2023-10-20T03:35:25.380 INFO:tasks.rados.rados.0.smithi170.stdout:123: finishing rollback tid 0 to smithi17039300-29 2023-10-20T03:35:25.381 INFO:tasks.rados.rados.0.smithi170.stdout:121: finishing write tid 5 to smithi17039300-31 2023-10-20T03:35:25.381 INFO:tasks.rados.rados.0.smithi170.stdout:121: oid 31 updating version 0 to 15 2023-10-20T03:35:25.382 INFO:tasks.rados.rados.0.smithi170.stdout:121: oid 31 updating version 15 to 17 2023-10-20T03:35:25.382 INFO:tasks.rados.rados.0.smithi170.stdout:121: oid 31 version 17 is already newer than 16 2023-10-20T03:35:25.382 INFO:tasks.rados.rados.0.smithi170.stdout:121: oid 31 version 17 is already newer than 14 2023-10-20T03:35:25.382 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 17 (ObjNum 78 snap 7 seq_num 78) dirty exists 2023-10-20T03:35:25.383 INFO:tasks.rados.rados.0.smithi170.stdout:121: left oid 31 (ObjNum 78 snap 7 seq_num 78) 2023-10-20T03:35:25.383 INFO:tasks.rados.rados.0.smithi170.stdout:123: finishing rollback tid 1 to smithi17039300-29 2023-10-20T03:35:25.383 INFO:tasks.rados.rados.0.smithi170.stdout:123: finishing rollback tid 2 to smithi17039300-29 2023-10-20T03:35:25.384 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 29 v 10 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-10-20T03:35:25.384 INFO:tasks.rados.rados.0.smithi170.stdout:122: finishing write tid 1 to smithi17039300-14 2023-10-20T03:35:25.395 INFO:tasks.rados.rados.0.smithi170.stdout:122: finishing write tid 2 to smithi17039300-14 2023-10-20T03:35:25.396 INFO:tasks.rados.rados.0.smithi170.stdout:122: finishing write tid 3 to smithi17039300-14 2023-10-20T03:35:25.397 INFO:tasks.rados.rados.0.smithi170.stdout:122: finishing write tid 4 to smithi17039300-14 2023-10-20T03:35:25.401 INFO:tasks.rados.rados.0.smithi170.stdout:122: finishing write tid 5 to smithi17039300-14 2023-10-20T03:35:25.401 INFO:tasks.rados.rados.0.smithi170.stdout:122: oid 14 updating version 0 to 24 2023-10-20T03:35:25.401 INFO:tasks.rados.rados.0.smithi170.stdout:122: oid 14 updating version 24 to 26 2023-10-20T03:35:25.402 INFO:tasks.rados.rados.0.smithi170.stdout:122: oid 14 version 26 is already newer than 25 2023-10-20T03:35:25.402 INFO:tasks.rados.rados.0.smithi170.stdout:122: oid 14 updating version 26 to 27 2023-10-20T03:35:25.402 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 27 (ObjNum 79 snap 7 seq_num 79) dirty exists 2023-10-20T03:35:25.402 INFO:tasks.rados.rados.0.smithi170.stdout:122: left oid 14 (ObjNum 79 snap 7 seq_num 79) 2023-10-20T03:35:25.420 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:25.417+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:25.436 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:25.437 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:25.437 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:25.437 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:35:25.437 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.438 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s2 36/31 ECSubWrite(tid=21, reqid=client.4570.0:198, at_version=36'13, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:25.438 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.265066+0000", 2023-10-20T03:35:25.438 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.1703675850000002, 2023-10-20T03:35:25.438 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.007859797, 2023-10-20T03:35:25.438 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.439 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.439 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.439 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.439 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.439 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265066+0000", 2023-10-20T03:35:25.440 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.440 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.440 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.441 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265066+0000", 2023-10-20T03:35:25.441 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.442 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.442 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.442 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.442 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265066+0000", 2023-10-20T03:35:25.443 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.35e-07 2023-10-20T03:35:25.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.443 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.444 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.444 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265361+0000", 2023-10-20T03:35:25.445 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00029424799999999999 2023-10-20T03:35:25.445 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.446 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.446 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.446 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265372+0000", 2023-10-20T03:35:25.447 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1834000000000001e-05 2023-10-20T03:35:25.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.447 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.447 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.448 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265382+0000", 2023-10-20T03:35:25.448 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4059999999999994e-06 2023-10-20T03:35:25.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.449 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.449 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.449 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265460+0000", 2023-10-20T03:35:25.449 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7844999999999998e-05 2023-10-20T03:35:25.450 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.450 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.450 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:25.450 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265467+0000", 2023-10-20T03:35:25.450 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7530000000000001e-06 2023-10-20T03:35:25.451 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.451 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.451 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:25.451 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.272862+0000", 2023-10-20T03:35:25.452 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073944290000000001 2023-10-20T03:35:25.452 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.452 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.452 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.453 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.272926+0000", 2023-10-20T03:35:25.453 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4146999999999994e-05 2023-10-20T03:35:25.453 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.453 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.454 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.454 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.454 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s5 36/31 ECSubWrite(tid=59, reqid=client.4570.0:202, at_version=36'16, trim_to=0'0, roll_forward_to=34'14))", 2023-10-20T03:35:25.454 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.275848+0000", 2023-10-20T03:35:25.455 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.159586129, 2023-10-20T03:35:25.455 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001989392, 2023-10-20T03:35:25.455 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.455 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.455 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.456 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.456 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275848+0000", 2023-10-20T03:35:25.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275848+0000", 2023-10-20T03:35:25.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.458 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275848+0000", 2023-10-20T03:35:25.459 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1899999999999998e-07 2023-10-20T03:35:25.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.459 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.459 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.459 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275875+0000", 2023-10-20T03:35:25.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6910000000000002e-05 2023-10-20T03:35:25.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.460 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.461 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275888+0000", 2023-10-20T03:35:25.461 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2838e-05 2023-10-20T03:35:25.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.461 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.462 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.462 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275898+0000", 2023-10-20T03:35:25.462 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.73e-06 2023-10-20T03:35:25.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.463 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.463 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275957+0000", 2023-10-20T03:35:25.463 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9768000000000001e-05 2023-10-20T03:35:25.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.464 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.464 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:25.464 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275965+0000", 2023-10-20T03:35:25.464 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6140000000000003e-06 2023-10-20T03:35:25.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.465 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.465 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:25.465 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.277796+0000", 2023-10-20T03:35:25.465 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0018305579999999999 2023-10-20T03:35:25.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.466 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.466 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.277837+0000", 2023-10-20T03:35:25.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1554999999999998e-05 2023-10-20T03:35:25.467 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.467 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.467 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.468 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s5 36/31 ECSubWrite(tid=60, reqid=client.4570.0:203, at_version=36'17, trim_to=0'0, roll_forward_to=34'14))", 2023-10-20T03:35:25.468 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.278217+0000", 2023-10-20T03:35:25.468 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.1572166040000003, 2023-10-20T03:35:25.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014205890000000001, 2023-10-20T03:35:25.469 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.469 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.469 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.469 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278217+0000", 2023-10-20T03:35:25.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278217+0000", 2023-10-20T03:35:25.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278218+0000", 2023-10-20T03:35:25.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9700000000000005e-07 2023-10-20T03:35:25.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278245+0000", 2023-10-20T03:35:25.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6627000000000001e-05 2023-10-20T03:35:25.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278257+0000", 2023-10-20T03:35:25.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1779e-05 2023-10-20T03:35:25.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278264+0000", 2023-10-20T03:35:25.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1439999999999997e-06 2023-10-20T03:35:25.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278325+0000", 2023-10-20T03:35:25.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1576999999999997e-05 2023-10-20T03:35:25.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:25.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.278334+0000", 2023-10-20T03:35:25.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1579999999999994e-06 2023-10-20T03:35:25.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.478 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:25.478 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279597+0000", 2023-10-20T03:35:25.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001262406 2023-10-20T03:35:25.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.479 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279638+0000", 2023-10-20T03:35:25.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1001000000000003e-05 2023-10-20T03:35:25.480 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.480 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.480 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.481 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s5 36/31 ECSubWrite(tid=61, reqid=client.4570.0:204, at_version=36'18, trim_to=0'0, roll_forward_to=34'14))", 2023-10-20T03:35:25.481 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.279679+0000", 2023-10-20T03:35:25.481 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.1557545080000002, 2023-10-20T03:35:25.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0025554420000000002, 2023-10-20T03:35:25.482 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.482 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.482 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279679+0000", 2023-10-20T03:35:25.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279679+0000", 2023-10-20T03:35:25.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.485 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.485 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.485 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279680+0000", 2023-10-20T03:35:25.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.99e-07 2023-10-20T03:35:25.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279702+0000", 2023-10-20T03:35:25.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2181999999999999e-05 2023-10-20T03:35:25.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279719+0000", 2023-10-20T03:35:25.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6532e-05 2023-10-20T03:35:25.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279728+0000", 2023-10-20T03:35:25.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0720000000000006e-06 2023-10-20T03:35:25.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279770+0000", 2023-10-20T03:35:25.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2400000000000001e-05 2023-10-20T03:35:25.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:25.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.279779+0000", 2023-10-20T03:35:25.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8230000000000001e-06 2023-10-20T03:35:25.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:25.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.282182+0000", 2023-10-20T03:35:25.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024034569999999999 2023-10-20T03:35:25.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.282235+0000", 2023-10-20T03:35:25.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2376999999999997e-05 2023-10-20T03:35:25.494 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.494 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.495 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.495 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:206 3.fs0 3:f289eea4:::smithi17039300-43:head [stat,delete] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-10-20T03:35:25.496 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.281026+0000", 2023-10-20T03:35:25.496 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.1544078740000003, 2023-10-20T03:35:25.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081465319999999997, 2023-10-20T03:35:25.496 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.497 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:25.497 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:25.497 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:25.497 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:25.497 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 206 2023-10-20T03:35:25.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.498 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281026+0000", 2023-10-20T03:35:25.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281026+0000", 2023-10-20T03:35:25.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281028+0000", 2023-10-20T03:35:25.501 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.73e-06 2023-10-20T03:35:25.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.501 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.501 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281034+0000", 2023-10-20T03:35:25.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7499999999999997e-06 2023-10-20T03:35:25.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281054+0000", 2023-10-20T03:35:25.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9601000000000001e-05 2023-10-20T03:35:25.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281066+0000", 2023-10-20T03:35:25.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1538e-05 2023-10-20T03:35:25.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281120+0000", 2023-10-20T03:35:25.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4268999999999998e-05 2023-10-20T03:35:25.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.505 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:25.505 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281193+0000", 2023-10-20T03:35:25.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2675999999999993e-05 2023-10-20T03:35:25.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:25.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281910+0000", 2023-10-20T03:35:25.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00071703499999999998 2023-10-20T03:35:25.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:25.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.283019+0000", 2023-10-20T03:35:25.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011095860000000001 2023-10-20T03:35:25.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.508 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:25.508 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.289127+0000", 2023-10-20T03:35:25.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00610796 2023-10-20T03:35:25.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.509 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.289172+0000", 2023-10-20T03:35:25.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5386999999999998e-05 2023-10-20T03:35:25.510 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.510 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.510 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.511 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s2 36/31 ECSubWrite(tid=22, reqid=client.4570.0:199, at_version=36'14, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:25.511 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.284788+0000", 2023-10-20T03:35:25.511 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.1506461530000003, 2023-10-20T03:35:25.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0040053809999999997, 2023-10-20T03:35:25.512 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.512 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.512 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.512 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.512 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.513 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284788+0000", 2023-10-20T03:35:25.513 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.513 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.513 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.513 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.514 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284788+0000", 2023-10-20T03:35:25.514 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.515 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.515 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284788+0000", 2023-10-20T03:35:25.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5700000000000002e-07 2023-10-20T03:35:25.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.516 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284902+0000", 2023-10-20T03:35:25.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000113083 2023-10-20T03:35:25.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.518 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284913+0000", 2023-10-20T03:35:25.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1736000000000001e-05 2023-10-20T03:35:25.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284919+0000", 2023-10-20T03:35:25.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1929999999999998e-06 2023-10-20T03:35:25.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284989+0000", 2023-10-20T03:35:25.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9439999999999999e-05 2023-10-20T03:35:25.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:25.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.285000+0000", 2023-10-20T03:35:25.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0821e-05 2023-10-20T03:35:25.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:25.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.288764+0000", 2023-10-20T03:35:25.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003763836 2023-10-20T03:35:25.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.288793+0000", 2023-10-20T03:35:25.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9515000000000001e-05 2023-10-20T03:35:25.523 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.524 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.524 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.525 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 36/31 ECSubWriteReply(tid=32, last_complete=36'13, committed=1, applied=1))", 2023-10-20T03:35:25.525 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.289011+0000", 2023-10-20T03:35:25.525 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.1464228829999996, 2023-10-20T03:35:25.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00048037600000000002, 2023-10-20T03:35:25.525 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.526 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.526 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.526 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.289011+0000", 2023-10-20T03:35:25.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.289011+0000", 2023-10-20T03:35:25.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.289012+0000", 2023-10-20T03:35:25.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2699999999999999e-07 2023-10-20T03:35:25.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.289015+0000", 2023-10-20T03:35:25.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5360000000000001e-06 2023-10-20T03:35:25.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.289025+0000", 2023-10-20T03:35:25.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5270000000000004e-06 2023-10-20T03:35:25.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.289030+0000", 2023-10-20T03:35:25.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9119999999999997e-06 2023-10-20T03:35:25.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.289077+0000", 2023-10-20T03:35:25.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7110000000000001e-05 2023-10-20T03:35:25.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.289491+0000", 2023-10-20T03:35:25.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00041466400000000003 2023-10-20T03:35:25.535 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.535 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.536 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.537 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s5 36/31 ECSubWrite(tid=62, reqid=client.4570.0:204, at_version=0'0, trim_to=0'0, roll_forward_to=36'18))", 2023-10-20T03:35:25.537 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.292141+0000", 2023-10-20T03:35:25.537 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.1432923869999998, 2023-10-20T03:35:25.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00067776100000000003, 2023-10-20T03:35:25.538 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.538 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.539 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292141+0000", 2023-10-20T03:35:25.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292141+0000", 2023-10-20T03:35:25.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292142+0000", 2023-10-20T03:35:25.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9800000000000004e-07 2023-10-20T03:35:25.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292154+0000", 2023-10-20T03:35:25.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1946999999999999e-05 2023-10-20T03:35:25.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292164+0000", 2023-10-20T03:35:25.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6719999999999999e-06 2023-10-20T03:35:25.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292169+0000", 2023-10-20T03:35:25.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1900000000000003e-06 2023-10-20T03:35:25.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292214+0000", 2023-10-20T03:35:25.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5553999999999998e-05 2023-10-20T03:35:25.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:25.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292219+0000", 2023-10-20T03:35:25.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9350000000000002e-06 2023-10-20T03:35:25.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:25.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292795+0000", 2023-10-20T03:35:25.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00057565100000000005 2023-10-20T03:35:25.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.292819+0000", 2023-10-20T03:35:25.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4314000000000001e-05 2023-10-20T03:35:25.552 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.552 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.552 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.553 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s2 36/31 ECSubWrite(tid=23, reqid=client.4570.0:200, at_version=36'15, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:25.553 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.293431+0000", 2023-10-20T03:35:25.553 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.1420030790000002, 2023-10-20T03:35:25.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00073801100000000001, 2023-10-20T03:35:25.554 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.554 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.554 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293431+0000", 2023-10-20T03:35:25.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293431+0000", 2023-10-20T03:35:25.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293431+0000", 2023-10-20T03:35:25.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0900000000000001e-07 2023-10-20T03:35:25.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293446+0000", 2023-10-20T03:35:25.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4595e-05 2023-10-20T03:35:25.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293457+0000", 2023-10-20T03:35:25.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0733999999999999e-05 2023-10-20T03:35:25.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293468+0000", 2023-10-20T03:35:25.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1602e-05 2023-10-20T03:35:25.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293522+0000", 2023-10-20T03:35:25.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3587e-05 2023-10-20T03:35:25.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.563 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:25.563 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293527+0000", 2023-10-20T03:35:25.563 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4909999999999996e-06 2023-10-20T03:35:25.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:25.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.294132+0000", 2023-10-20T03:35:25.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00060423300000000003 2023-10-20T03:35:25.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.294169+0000", 2023-10-20T03:35:25.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7160000000000003e-05 2023-10-20T03:35:25.566 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.566 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.566 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.567 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s2 36/31 ECSubWrite(tid=24, reqid=client.4570.0:200, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-10-20T03:35:25.567 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.297775+0000", 2023-10-20T03:35:25.567 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.1376584919999999, 2023-10-20T03:35:25.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081359300000000004, 2023-10-20T03:35:25.568 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.568 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.568 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297775+0000", 2023-10-20T03:35:25.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297775+0000", 2023-10-20T03:35:25.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297776+0000", 2023-10-20T03:35:25.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8500000000000001e-07 2023-10-20T03:35:25.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.572 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297785+0000", 2023-10-20T03:35:25.572 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4190000000000008e-06 2023-10-20T03:35:25.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297797+0000", 2023-10-20T03:35:25.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1941e-05 2023-10-20T03:35:25.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.574 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.574 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297803+0000", 2023-10-20T03:35:25.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6670000000000001e-06 2023-10-20T03:35:25.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297850+0000", 2023-10-20T03:35:25.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6755999999999998e-05 2023-10-20T03:35:25.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.576 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:25.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297855+0000", 2023-10-20T03:35:25.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5350000000000004e-06 2023-10-20T03:35:25.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:25.578 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.298538+0000", 2023-10-20T03:35:25.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00068245399999999998 2023-10-20T03:35:25.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.579 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.579 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.298589+0000", 2023-10-20T03:35:25.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.1236000000000002e-05 2023-10-20T03:35:25.579 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.579 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.580 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.580 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.7s6 38/31 ECSubWrite(tid=63, reqid=client.4570.0:216, at_version=38'7, trim_to=0'0, roll_forward_to=34'5))", 2023-10-20T03:35:25.580 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:20.272713+0000", 2023-10-20T03:35:25.581 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.1627206760000002, 2023-10-20T03:35:25.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002332382, 2023-10-20T03:35:25.581 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.581 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.581 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.272713+0000", 2023-10-20T03:35:25.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.583 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.583 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.272713+0000", 2023-10-20T03:35:25.583 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.583 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.584 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.584 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.584 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.272713+0000", 2023-10-20T03:35:25.584 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9200000000000002e-07 2023-10-20T03:35:25.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.585 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.585 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.585 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.272751+0000", 2023-10-20T03:35:25.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7567000000000001e-05 2023-10-20T03:35:25.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.586 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.586 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.586 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.272781+0000", 2023-10-20T03:35:25.587 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0192000000000002e-05 2023-10-20T03:35:25.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.587 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.587 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.587 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.272796+0000", 2023-10-20T03:35:25.588 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5078e-05 2023-10-20T03:35:25.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.272890+0000", 2023-10-20T03:35:25.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4123000000000007e-05 2023-10-20T03:35:25.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.589 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:25.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.272902+0000", 2023-10-20T03:35:25.590 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1324e-05 2023-10-20T03:35:25.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:25.591 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.275002+0000", 2023-10-20T03:35:25.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.002099779 2023-10-20T03:35:25.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.592 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.592 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.275046+0000", 2023-10-20T03:35:25.592 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4026999999999997e-05 2023-10-20T03:35:25.592 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.592 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.593 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.593 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.7s6 38/31 ECSubWrite(tid=64, reqid=client.4570.0:216, at_version=0'0, trim_to=0'0, roll_forward_to=38'7))", 2023-10-20T03:35:25.593 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:20.278411+0000", 2023-10-20T03:35:25.594 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.1570230529999996, 2023-10-20T03:35:25.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00096839199999999999, 2023-10-20T03:35:25.594 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.594 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.595 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.595 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.595 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.278411+0000", 2023-10-20T03:35:25.596 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.596 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.596 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.596 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.278411+0000", 2023-10-20T03:35:25.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.278412+0000", 2023-10-20T03:35:25.598 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1179999999999999e-06 2023-10-20T03:35:25.598 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.598 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.598 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.599 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.278457+0000", 2023-10-20T03:35:25.599 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5229999999999999e-05 2023-10-20T03:35:25.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.599 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.599 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.600 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.278475+0000", 2023-10-20T03:35:25.600 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7679000000000001e-05 2023-10-20T03:35:25.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.601 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.601 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.278486+0000", 2023-10-20T03:35:25.601 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0879999999999999e-05 2023-10-20T03:35:25.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.601 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.602 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.602 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.278545+0000", 2023-10-20T03:35:25.602 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9719999999999997e-05 2023-10-20T03:35:25.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.604 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:25.604 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.278555+0000", 2023-10-20T03:35:25.604 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4169999999999998e-06 2023-10-20T03:35:25.605 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.605 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.606 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:25.606 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.279339+0000", 2023-10-20T03:35:25.606 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00078467800000000002 2023-10-20T03:35:25.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:20.279379+0000", 2023-10-20T03:35:25.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9669999999999998e-05 2023-10-20T03:35:25.608 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.609 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.609 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.609 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(osd.13.22:1 3.fs0 3:f289eea4:::smithi17039300-43:head [assert-version v15,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-10-20T03:35:25.610 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:23.448982+0000", 2023-10-20T03:35:25.610 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 1.986451752, 2023-10-20T03:35:25.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00514052, 2023-10-20T03:35:25.610 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.611 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:25.611 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.611 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.611 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.448982+0000", 2023-10-20T03:35:25.612 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.612 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.612 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.612 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.613 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.448982+0000", 2023-10-20T03:35:25.613 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.448985+0000", 2023-10-20T03:35:25.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.509e-06 2023-10-20T03:35:25.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.615 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.449012+0000", 2023-10-20T03:35:25.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7477000000000001e-05 2023-10-20T03:35:25.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.449046+0000", 2023-10-20T03:35:25.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3458999999999999e-05 2023-10-20T03:35:25.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.617 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.617 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.449061+0000", 2023-10-20T03:35:25.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5752000000000001e-05 2023-10-20T03:35:25.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.618 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.449205+0000", 2023-10-20T03:35:25.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014325300000000001 2023-10-20T03:35:25.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:25.620 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.449368+0000", 2023-10-20T03:35:25.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016306299999999999 2023-10-20T03:35:25.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.620 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.620 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.621 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.454123+0000", 2023-10-20T03:35:25.621 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047550070000000003 2023-10-20T03:35:25.621 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.621 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.621 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.622 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpRead(3.fs0 41/31 ECSubRead(tid=57, to_read={3:f289eea4:::smithi17039300-43:head=0,200704,0}, subchunks={3:f289eea4:::smithi17039300-43:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:25.622 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:23.449744+0000", 2023-10-20T03:35:25.623 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 1.985689464, 2023-10-20T03:35:25.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0017025129999999999, 2023-10-20T03:35:25.623 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.623 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.623 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.624 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.624 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.449744+0000", 2023-10-20T03:35:25.624 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.625 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.449744+0000", 2023-10-20T03:35:25.625 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.626 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.626 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.449744+0000", 2023-10-20T03:35:25.626 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.627 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.627 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.449744+0000", 2023-10-20T03:35:25.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.449778+0000", 2023-10-20T03:35:25.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3929e-05 2023-10-20T03:35:25.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.630 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.449797+0000", 2023-10-20T03:35:25.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8992999999999999e-05 2023-10-20T03:35:25.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.631 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.450108+0000", 2023-10-20T03:35:25.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031115199999999998 2023-10-20T03:35:25.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451447+0000", 2023-10-20T03:35:25.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013384390000000001 2023-10-20T03:35:25.633 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.634 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.634 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.635 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.635 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpReadReply(3.fs0 41/31 ECSubReadReply(tid=57, attrs_read=0))", 2023-10-20T03:35:25.636 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:23.450612+0000", 2023-10-20T03:35:25.636 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 1.984821704, 2023-10-20T03:35:25.636 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010235050000000001, 2023-10-20T03:35:25.636 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.637 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.637 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.450612+0000", 2023-10-20T03:35:25.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.639 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.639 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.450612+0000", 2023-10-20T03:35:25.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.640 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.450614+0000", 2023-10-20T03:35:25.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.59e-06 2023-10-20T03:35:25.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451072+0000", 2023-10-20T03:35:25.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00045829300000000003 2023-10-20T03:35:25.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.642 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451115+0000", 2023-10-20T03:35:25.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2809000000000003e-05 2023-10-20T03:35:25.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.643 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451139+0000", 2023-10-20T03:35:25.644 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4456000000000001e-05 2023-10-20T03:35:25.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.644 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.644 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451518+0000", 2023-10-20T03:35:25.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00037853000000000002 2023-10-20T03:35:25.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451636+0000", 2023-10-20T03:35:25.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117827 2023-10-20T03:35:25.646 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.646 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.646 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.647 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.647 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpReadReply(3.fs0 41/31 ECSubReadReply(tid=57, attrs_read=0))", 2023-10-20T03:35:25.647 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:23.451262+0000", 2023-10-20T03:35:25.647 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 1.9841720819999999, 2023-10-20T03:35:25.648 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00064810499999999995, 2023-10-20T03:35:25.648 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.648 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.648 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.648 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451262+0000", 2023-10-20T03:35:25.649 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451262+0000", 2023-10-20T03:35:25.650 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.651 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.651 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451263+0000", 2023-10-20T03:35:25.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2100000000000001e-06 2023-10-20T03:35:25.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.652 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.652 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451525+0000", 2023-10-20T03:35:25.653 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026184899999999998 2023-10-20T03:35:25.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.653 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.653 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.654 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451555+0000", 2023-10-20T03:35:25.654 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0349e-05 2023-10-20T03:35:25.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.654 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.655 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451571+0000", 2023-10-20T03:35:25.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5886e-05 2023-10-20T03:35:25.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.655 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451850+0000", 2023-10-20T03:35:25.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00027908300000000001 2023-10-20T03:35:25.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.657 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.657 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451910+0000", 2023-10-20T03:35:25.657 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.9728000000000001e-05 2023-10-20T03:35:25.657 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.658 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.658 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.658 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpReadReply(3.fs0 41/31 ECSubReadReply(tid=57, attrs_read=0))", 2023-10-20T03:35:25.659 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:23.451429+0000", 2023-10-20T03:35:25.659 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 1.9840050310000001, 2023-10-20T03:35:25.659 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00035852399999999998, 2023-10-20T03:35:25.659 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.659 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.660 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.660 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.660 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.660 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451429+0000", 2023-10-20T03:35:25.660 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.661 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.661 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451429+0000", 2023-10-20T03:35:25.661 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.662 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.662 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451429+0000", 2023-10-20T03:35:25.663 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.663 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.663 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451429+0000", 2023-10-20T03:35:25.664 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.664 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.664 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.665 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.665 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451462+0000", 2023-10-20T03:35:25.665 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3497000000000001e-05 2023-10-20T03:35:25.666 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.666 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.666 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.667 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451473+0000", 2023-10-20T03:35:25.667 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0538e-05 2023-10-20T03:35:25.667 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.667 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.668 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.668 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451716+0000", 2023-10-20T03:35:25.668 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00024296100000000001 2023-10-20T03:35:25.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.669 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.669 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.451787+0000", 2023-10-20T03:35:25.669 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1527999999999997e-05 2023-10-20T03:35:25.669 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.670 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.670 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.670 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.670 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpReadReply(3.fs0 40/31 ECSubReadReply(tid=57, attrs_read=0))", 2023-10-20T03:35:25.671 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:23.452533+0000", 2023-10-20T03:35:25.671 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 1.9829011940000001, 2023-10-20T03:35:25.671 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0016342970000000001, 2023-10-20T03:35:25.671 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.671 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.672 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.672 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.672 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.672 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.452533+0000", 2023-10-20T03:35:25.673 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.673 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.673 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.673 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.673 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.452533+0000", 2023-10-20T03:35:25.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.674 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.452534+0000", 2023-10-20T03:35:25.675 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.553e-06 2023-10-20T03:35:25.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.676 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.453271+0000", 2023-10-20T03:35:25.676 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00073718600000000003 2023-10-20T03:35:25.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.676 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.677 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.453300+0000", 2023-10-20T03:35:25.677 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8819e-05 2023-10-20T03:35:25.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.677 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.677 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.678 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.453317+0000", 2023-10-20T03:35:25.678 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6557000000000001e-05 2023-10-20T03:35:25.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.679 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.679 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.453434+0000", 2023-10-20T03:35:25.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000117433 2023-10-20T03:35:25.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.679 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.680 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.680 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.454167+0000", 2023-10-20T03:35:25.680 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00073274899999999999 2023-10-20T03:35:25.680 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.680 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.681 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.681 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpRead(3.2s2 41/31 ECSubRead(tid=26, to_read={3:474cd252:::smithi17039300-42:head=0,430080,0}, subchunks={3:474cd252:::smithi17039300-42:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:25.681 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:23.455511+0000", 2023-10-20T03:35:25.682 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 1.979922714, 2023-10-20T03:35:25.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013011909999999999, 2023-10-20T03:35:25.682 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.683 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.683 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.683 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.455511+0000", 2023-10-20T03:35:25.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.685 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.685 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.685 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.455511+0000", 2023-10-20T03:35:25.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.686 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.455512+0000", 2023-10-20T03:35:25.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3370000000000001e-06 2023-10-20T03:35:25.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.688 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.455530+0000", 2023-10-20T03:35:25.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7082999999999999e-05 2023-10-20T03:35:25.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.688 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.455562+0000", 2023-10-20T03:35:25.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2839000000000002e-05 2023-10-20T03:35:25.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.690 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.455580+0000", 2023-10-20T03:35:25.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8023e-05 2023-10-20T03:35:25.690 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.691 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.455711+0000", 2023-10-20T03:35:25.691 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000130474 2023-10-20T03:35:25.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.692 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.692 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.456812+0000", 2023-10-20T03:35:25.692 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001101435 2023-10-20T03:35:25.692 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.693 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.693 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.693 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.693 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpRead(3.2s2 41/31 ECSubRead(tid=28, to_read={3:474cd252:::smithi17039300-42:head=0,430080,0}, subchunks={3:474cd252:::smithi17039300-42:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:25.694 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:23.458932+0000", 2023-10-20T03:35:25.694 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 1.976501821, 2023-10-20T03:35:25.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000367742, 2023-10-20T03:35:25.694 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:25.694 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:25.695 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:25.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:25.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.458932+0000", 2023-10-20T03:35:25.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:25.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.458932+0000", 2023-10-20T03:35:25.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:25.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:25.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.458933+0000", 2023-10-20T03:35:25.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.223e-06 2023-10-20T03:35:25.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:25.699 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.458950+0000", 2023-10-20T03:35:25.699 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6818999999999999e-05 2023-10-20T03:35:25.699 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.699 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.699 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:25.700 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.458999+0000", 2023-10-20T03:35:25.700 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9279e-05 2023-10-20T03:35:25.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.700 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:25.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.459026+0000", 2023-10-20T03:35:25.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6948999999999999e-05 2023-10-20T03:35:25.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:25.702 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.459171+0000", 2023-10-20T03:35:25.702 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014465600000000001 2023-10-20T03:35:25.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:25.702 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:25.702 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:25.703 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:23.459300+0000", 2023-10-20T03:35:25.703 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012881599999999999 2023-10-20T03:35:25.703 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.703 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.703 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.704 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:25.704 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:25.704 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:25.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-10-20T03:35:25.706 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:25.705+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:25.807 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:25.806+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:25.878 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:25.878 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:25.878 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:25.879 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:25.892 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-10-20T03:35:25.942 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:25.941+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:26.008 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:26.005+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:26.020 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:26.020 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:26.020 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:35:26.021 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:35:26.021 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:26.035 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-10-20T03:35:26.109 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:26.108+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:26.199 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:26.199 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:26.200 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:26.200 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:35:26.200 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.200 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s3 36/31 ECSubWrite(tid=21, reqid=client.4570.0:198, at_version=36'13, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:26.200 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.265679+0000", 2023-10-20T03:35:26.201 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.9310609300000001, 2023-10-20T03:35:26.201 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0082907580000000005, 2023-10-20T03:35:26.201 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.201 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.202 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.202 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.202 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.202 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265679+0000", 2023-10-20T03:35:26.202 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.203 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.203 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.203 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.203 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265679+0000", 2023-10-20T03:35:26.204 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.204 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.204 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.204 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.204 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265680+0000", 2023-10-20T03:35:26.205 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.04e-07 2023-10-20T03:35:26.205 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.205 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.205 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.205 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266116+0000", 2023-10-20T03:35:26.206 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00043625499999999998 2023-10-20T03:35:26.206 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.206 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.206 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.206 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266135+0000", 2023-10-20T03:35:26.206 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8974000000000002e-05 2023-10-20T03:35:26.207 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.207 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.207 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.207 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266146+0000", 2023-10-20T03:35:26.207 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1523e-05 2023-10-20T03:35:26.208 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.208 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.208 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.208 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266263+0000", 2023-10-20T03:35:26.208 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000116945 2023-10-20T03:35:26.209 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.209 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.209 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.209 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266275+0000", 2023-10-20T03:35:26.209 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1941e-05 2023-10-20T03:35:26.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.210 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.210 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.210 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.273944+0000", 2023-10-20T03:35:26.210 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0076690689999999997 2023-10-20T03:35:26.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.211 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.211 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.273970+0000", 2023-10-20T03:35:26.211 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5846999999999998e-05 2023-10-20T03:35:26.212 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.212 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.212 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.212 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.213 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.es3 36/31 ECSubWrite(tid=21, reqid=client.4570.0:205, at_version=36'12, trim_to=0'0, roll_forward_to=34'9))", 2023-10-20T03:35:26.213 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.270711+0000", 2023-10-20T03:35:26.213 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.9260287659999999, 2023-10-20T03:35:26.213 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014804320000000001, 2023-10-20T03:35:26.213 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.213 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.214 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.214 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.214 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.214 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.270711+0000", 2023-10-20T03:35:26.214 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.215 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.215 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.215 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.215 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.270711+0000", 2023-10-20T03:35:26.215 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.216 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.216 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.216 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.270712+0000", 2023-10-20T03:35:26.216 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8500000000000002e-07 2023-10-20T03:35:26.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.217 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.217 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.217 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.270745+0000", 2023-10-20T03:35:26.217 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2957999999999999e-05 2023-10-20T03:35:26.217 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.218 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.218 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.218 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.270759+0000", 2023-10-20T03:35:26.218 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4635e-05 2023-10-20T03:35:26.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.219 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.219 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.270770+0000", 2023-10-20T03:35:26.219 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0504e-05 2023-10-20T03:35:26.219 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.220 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.220 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.220 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.270842+0000", 2023-10-20T03:35:26.220 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1933000000000004e-05 2023-10-20T03:35:26.220 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.221 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.221 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.270851+0000", 2023-10-20T03:35:26.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7059999999999999e-06 2023-10-20T03:35:26.221 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.272149+0000", 2023-10-20T03:35:26.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0012973819999999999 2023-10-20T03:35:26.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.272192+0000", 2023-10-20T03:35:26.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3028999999999997e-05 2023-10-20T03:35:26.223 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.223 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.224 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.224 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.es3 36/31 ECSubWrite(tid=22, reqid=client.4570.0:205, at_version=0'0, trim_to=0'0, roll_forward_to=36'12))", 2023-10-20T03:35:26.224 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.281378+0000", 2023-10-20T03:35:26.224 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.9153624919999999, 2023-10-20T03:35:26.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00116653, 2023-10-20T03:35:26.225 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.225 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.225 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.226 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.226 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281378+0000", 2023-10-20T03:35:26.226 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.226 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.226 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.227 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.227 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281378+0000", 2023-10-20T03:35:26.227 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.227 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281379+0000", 2023-10-20T03:35:26.228 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.46e-06 2023-10-20T03:35:26.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281400+0000", 2023-10-20T03:35:26.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1169e-05 2023-10-20T03:35:26.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.230 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281420+0000", 2023-10-20T03:35:26.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9378e-05 2023-10-20T03:35:26.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.230 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281431+0000", 2023-10-20T03:35:26.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0934999999999999e-05 2023-10-20T03:35:26.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281501+0000", 2023-10-20T03:35:26.232 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9983999999999999e-05 2023-10-20T03:35:26.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.232 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.232 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281507+0000", 2023-10-20T03:35:26.233 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3770000000000001e-06 2023-10-20T03:35:26.233 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.233 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.233 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.233 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.282503+0000", 2023-10-20T03:35:26.234 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00099557899999999991 2023-10-20T03:35:26.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.234 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.234 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.282544+0000", 2023-10-20T03:35:26.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1647999999999998e-05 2023-10-20T03:35:26.235 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.235 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.235 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.236 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s3 36/31 ECSubWrite(tid=22, reqid=client.4570.0:199, at_version=36'14, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:26.236 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.283633+0000", 2023-10-20T03:35:26.236 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.9131074290000001, 2023-10-20T03:35:26.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041089469999999999, 2023-10-20T03:35:26.237 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.237 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.237 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.283633+0000", 2023-10-20T03:35:26.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.283633+0000", 2023-10-20T03:35:26.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.283634+0000", 2023-10-20T03:35:26.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3300000000000001e-07 2023-10-20T03:35:26.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.241 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.283778+0000", 2023-10-20T03:35:26.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000144289 2023-10-20T03:35:26.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.241 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.283790+0000", 2023-10-20T03:35:26.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1810999999999999e-05 2023-10-20T03:35:26.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.283798+0000", 2023-10-20T03:35:26.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1810000000000001e-06 2023-10-20T03:35:26.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.283859+0000", 2023-10-20T03:35:26.244 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0751999999999997e-05 2023-10-20T03:35:26.244 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.244 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.244 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.283865+0000", 2023-10-20T03:35:26.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3289999999999999e-06 2023-10-20T03:35:26.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.245 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.287704+0000", 2023-10-20T03:35:26.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0038388379999999998 2023-10-20T03:35:26.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.287742+0000", 2023-10-20T03:35:26.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8013999999999999e-05 2023-10-20T03:35:26.247 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.247 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.247 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.247 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.248 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s3 36/31 ECSubWrite(tid=23, reqid=client.4570.0:200, at_version=36'15, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:26.248 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.293536+0000", 2023-10-20T03:35:26.248 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.9032037449999999, 2023-10-20T03:35:26.248 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00083306099999999996, 2023-10-20T03:35:26.249 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.249 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.249 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.249 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.249 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293536+0000", 2023-10-20T03:35:26.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.250 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.251 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293536+0000", 2023-10-20T03:35:26.251 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.251 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.251 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.252 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293537+0000", 2023-10-20T03:35:26.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1200000000000002e-07 2023-10-20T03:35:26.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.252 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.252 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293556+0000", 2023-10-20T03:35:26.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8300000000000001e-05 2023-10-20T03:35:26.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.253 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.254 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293570+0000", 2023-10-20T03:35:26.254 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4158000000000001e-05 2023-10-20T03:35:26.254 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.254 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.254 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.255 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293577+0000", 2023-10-20T03:35:26.255 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9419999999999998e-06 2023-10-20T03:35:26.255 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.255 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.255 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.256 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293638+0000", 2023-10-20T03:35:26.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1317999999999998e-05 2023-10-20T03:35:26.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.257 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.293648+0000", 2023-10-20T03:35:26.257 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0054000000000001e-05 2023-10-20T03:35:26.257 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.257 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.257 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.258 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.294344+0000", 2023-10-20T03:35:26.258 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00069581700000000005 2023-10-20T03:35:26.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.258 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.258 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.259 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.294370+0000", 2023-10-20T03:35:26.259 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5660000000000002e-05 2023-10-20T03:35:26.259 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.259 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.259 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.260 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.260 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s3 36/31 ECSubWrite(tid=24, reqid=client.4570.0:200, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-10-20T03:35:26.260 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.297820+0000", 2023-10-20T03:35:26.260 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.8989205340000002, 2023-10-20T03:35:26.261 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00076965400000000004, 2023-10-20T03:35:26.261 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.261 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.261 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.261 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.262 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.262 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297820+0000", 2023-10-20T03:35:26.262 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.262 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.262 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.263 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.263 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297820+0000", 2023-10-20T03:35:26.263 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.263 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.263 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.263 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.264 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297821+0000", 2023-10-20T03:35:26.264 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.032e-06 2023-10-20T03:35:26.264 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.264 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.264 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.265 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297837+0000", 2023-10-20T03:35:26.265 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6174000000000001e-05 2023-10-20T03:35:26.265 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.265 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.265 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.266 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297854+0000", 2023-10-20T03:35:26.266 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.755e-05 2023-10-20T03:35:26.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.266 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.266 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.266 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297864+0000", 2023-10-20T03:35:26.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8069999999999994e-06 2023-10-20T03:35:26.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297927+0000", 2023-10-20T03:35:26.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3163000000000005e-05 2023-10-20T03:35:26.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.297938+0000", 2023-10-20T03:35:26.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0158e-05 2023-10-20T03:35:26.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.269 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.298551+0000", 2023-10-20T03:35:26.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00061317999999999995 2023-10-20T03:35:26.270 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.270 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.270 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.270 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.298589+0000", 2023-10-20T03:35:26.270 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8590000000000002e-05 2023-10-20T03:35:26.271 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.271 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.271 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.272 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpRead(3.2s3 36/31 ECSubRead(tid=25, to_read={3:474cd252:::smithi17039300-42:head=0,4096,0}, subchunks={3:474cd252:::smithi17039300-42:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:26.272 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.299120+0000", 2023-10-20T03:35:26.272 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.8976197619999997, 2023-10-20T03:35:26.272 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000178775, 2023-10-20T03:35:26.272 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.273 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.273 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.299120+0000", 2023-10-20T03:35:26.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.299120+0000", 2023-10-20T03:35:26.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.299121+0000", 2023-10-20T03:35:26.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9700000000000005e-07 2023-10-20T03:35:26.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.276 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.276 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.299131+0000", 2023-10-20T03:35:26.276 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7310000000000005e-06 2023-10-20T03:35:26.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.277 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.277 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.299152+0000", 2023-10-20T03:35:26.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0801e-05 2023-10-20T03:35:26.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.299165+0000", 2023-10-20T03:35:26.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3472000000000001e-05 2023-10-20T03:35:26.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.299228+0000", 2023-10-20T03:35:26.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2730000000000004e-05 2023-10-20T03:35:26.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.299299+0000", 2023-10-20T03:35:26.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1143999999999995e-05 2023-10-20T03:35:26.280 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.281 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.281 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.281 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=23, reqid=client.4570.0:208, at_version=37'14, trim_to=0'0, roll_forward_to=36'12))", 2023-10-20T03:35:26.281 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.274891+0000", 2023-10-20T03:35:26.282 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.9218491970000002, 2023-10-20T03:35:26.282 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0066319839999999996, 2023-10-20T03:35:26.282 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.282 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.282 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.283 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.283 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.274891+0000", 2023-10-20T03:35:26.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.274891+0000", 2023-10-20T03:35:26.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.274892+0000", 2023-10-20T03:35:26.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5000000000000002e-07 2023-10-20T03:35:26.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.275312+0000", 2023-10-20T03:35:26.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00042037699999999999 2023-10-20T03:35:26.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.275323+0000", 2023-10-20T03:35:26.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0511e-05 2023-10-20T03:35:26.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.288 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.288 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.275332+0000", 2023-10-20T03:35:26.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9779999999999994e-06 2023-10-20T03:35:26.288 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.288 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.289 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.289 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.275408+0000", 2023-10-20T03:35:26.289 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6421000000000006e-05 2023-10-20T03:35:26.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.289 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.275416+0000", 2023-10-20T03:35:26.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1349999999999992e-06 2023-10-20T03:35:26.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.281464+0000", 2023-10-20T03:35:26.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0060482729999999998 2023-10-20T03:35:26.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.281523+0000", 2023-10-20T03:35:26.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8539000000000003e-05 2023-10-20T03:35:26.292 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.292 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.292 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.293 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=24, reqid=client.4570.0:209, at_version=37'15, trim_to=0'0, roll_forward_to=36'12))", 2023-10-20T03:35:26.293 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.289440+0000", 2023-10-20T03:35:26.293 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.907300577, 2023-10-20T03:35:26.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0046767390000000001, 2023-10-20T03:35:26.294 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.294 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.294 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.294 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.294 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.295 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.289440+0000", 2023-10-20T03:35:26.295 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.295 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.295 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.296 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.289440+0000", 2023-10-20T03:35:26.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.296 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.289441+0000", 2023-10-20T03:35:26.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9099999999999991e-07 2023-10-20T03:35:26.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.289557+0000", 2023-10-20T03:35:26.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000116066 2023-10-20T03:35:26.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.289574+0000", 2023-10-20T03:35:26.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7139999999999999e-05 2023-10-20T03:35:26.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.300 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.289582+0000", 2023-10-20T03:35:26.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8460000000000008e-06 2023-10-20T03:35:26.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.300 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.300 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.289653+0000", 2023-10-20T03:35:26.301 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1019000000000006e-05 2023-10-20T03:35:26.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.301 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.301 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.289661+0000", 2023-10-20T03:35:26.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5429999999999994e-06 2023-10-20T03:35:26.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.294059+0000", 2023-10-20T03:35:26.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043975220000000001 2023-10-20T03:35:26.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.294116+0000", 2023-10-20T03:35:26.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7612000000000003e-05 2023-10-20T03:35:26.304 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.304 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.304 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.305 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=25, reqid=client.4570.0:209, at_version=0'0, trim_to=0'0, roll_forward_to=37'15))", 2023-10-20T03:35:26.305 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.300432+0000", 2023-10-20T03:35:26.305 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.8963080559999996, 2023-10-20T03:35:26.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001266575, 2023-10-20T03:35:26.305 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.306 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.306 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.306 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.306 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.300432+0000", 2023-10-20T03:35:26.307 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.307 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.307 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.300432+0000", 2023-10-20T03:35:26.307 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.300433+0000", 2023-10-20T03:35:26.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.091e-06 2023-10-20T03:35:26.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.300453+0000", 2023-10-20T03:35:26.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9286000000000001e-05 2023-10-20T03:35:26.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.300466+0000", 2023-10-20T03:35:26.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3162999999999999e-05 2023-10-20T03:35:26.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.300473+0000", 2023-10-20T03:35:26.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5490000000000001e-06 2023-10-20T03:35:26.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.312 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.300580+0000", 2023-10-20T03:35:26.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000106436 2023-10-20T03:35:26.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.313 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.313 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.300593+0000", 2023-10-20T03:35:26.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3274e-05 2023-10-20T03:35:26.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.301650+0000", 2023-10-20T03:35:26.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010571879999999999 2023-10-20T03:35:26.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.301699+0000", 2023-10-20T03:35:26.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8588000000000003e-05 2023-10-20T03:35:26.316 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.316 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.316 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.316 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=26, reqid=client.4570.0:210, at_version=37'16, trim_to=0'0, roll_forward_to=37'15))", 2023-10-20T03:35:26.317 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.301709+0000", 2023-10-20T03:35:26.317 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.89503164, 2023-10-20T03:35:26.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00084419300000000003, 2023-10-20T03:35:26.317 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.317 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.318 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.301709+0000", 2023-10-20T03:35:26.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.319 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.319 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.319 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.301709+0000", 2023-10-20T03:35:26.319 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.320 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.301709+0000", 2023-10-20T03:35:26.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3399999999999998e-07 2023-10-20T03:35:26.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.301729+0000", 2023-10-20T03:35:26.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9627e-05 2023-10-20T03:35:26.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.301740+0000", 2023-10-20T03:35:26.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1336e-05 2023-10-20T03:35:26.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.301746+0000", 2023-10-20T03:35:26.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1539999999999999e-06 2023-10-20T03:35:26.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.324 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.301805+0000", 2023-10-20T03:35:26.324 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8770999999999997e-05 2023-10-20T03:35:26.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.325 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.325 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.301812+0000", 2023-10-20T03:35:26.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.754e-06 2023-10-20T03:35:26.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.326 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.326 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.302492+0000", 2023-10-20T03:35:26.326 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00068043200000000002 2023-10-20T03:35:26.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.327 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.302553+0000", 2023-10-20T03:35:26.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0384999999999999e-05 2023-10-20T03:35:26.327 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.328 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.328 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.328 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=27, reqid=client.4570.0:210, at_version=0'0, trim_to=0'0, roll_forward_to=37'16))", 2023-10-20T03:35:26.329 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.305866+0000", 2023-10-20T03:35:26.329 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.8908738200000004, 2023-10-20T03:35:26.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00066490500000000003, 2023-10-20T03:35:26.329 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.329 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.330 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.305866+0000", 2023-10-20T03:35:26.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.331 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.331 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.305866+0000", 2023-10-20T03:35:26.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.305867+0000", 2023-10-20T03:35:26.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0200000000000001e-07 2023-10-20T03:35:26.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.305883+0000", 2023-10-20T03:35:26.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6092000000000001e-05 2023-10-20T03:35:26.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.334 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.305894+0000", 2023-10-20T03:35:26.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1086e-05 2023-10-20T03:35:26.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.305900+0000", 2023-10-20T03:35:26.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8599999999999998e-06 2023-10-20T03:35:26.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.336 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.336 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.305999+0000", 2023-10-20T03:35:26.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8696999999999994e-05 2023-10-20T03:35:26.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.337 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.337 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.306008+0000", 2023-10-20T03:35:26.337 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4919999999999999e-06 2023-10-20T03:35:26.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.338 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.338 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.306493+0000", 2023-10-20T03:35:26.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00048423299999999998 2023-10-20T03:35:26.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.339 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.339 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.306531+0000", 2023-10-20T03:35:26.339 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8742999999999998e-05 2023-10-20T03:35:26.339 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.340 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.340 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.340 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpRead(3.es3 37/31 ECSubRead(tid=28, to_read={3:7dffa048:::smithi17039300-5:head=0,4096,0}, subchunks={3:7dffa048:::smithi17039300-5:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:26.341 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.306806+0000", 2023-10-20T03:35:26.341 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 6.8899343240000004, 2023-10-20T03:35:26.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022276500000000001, 2023-10-20T03:35:26.341 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.341 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.342 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.342 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.342 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.306806+0000", 2023-10-20T03:35:26.342 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.343 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.343 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.306806+0000", 2023-10-20T03:35:26.343 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.344 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.344 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.344 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.306806+0000", 2023-10-20T03:35:26.344 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7599999999999997e-07 2023-10-20T03:35:26.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.345 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.345 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.345 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.306814+0000", 2023-10-20T03:35:26.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6229999999999997e-06 2023-10-20T03:35:26.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.346 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.346 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.306826+0000", 2023-10-20T03:35:26.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1637000000000001e-05 2023-10-20T03:35:26.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.347 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.306832+0000", 2023-10-20T03:35:26.347 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5509999999999996e-06 2023-10-20T03:35:26.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.348 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.348 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.306938+0000", 2023-10-20T03:35:26.348 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010546400000000001 2023-10-20T03:35:26.349 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.349 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.349 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.349 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.307029+0000", 2023-10-20T03:35:26.349 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0914000000000004e-05 2023-10-20T03:35:26.349 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.350 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.350 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.350 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.350 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.350 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.1s1 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:26.350 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.329273+0000", 2023-10-20T03:35:26.351 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.86746718700000003, 2023-10-20T03:35:26.351 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049887400000000002, 2023-10-20T03:35:26.351 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.351 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.351 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.352 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.352 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.352 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.329273+0000", 2023-10-20T03:35:26.352 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.352 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.352 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.353 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.353 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.329273+0000", 2023-10-20T03:35:26.353 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.354 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.329273+0000", 2023-10-20T03:35:26.354 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.34e-07 2023-10-20T03:35:26.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.329685+0000", 2023-10-20T03:35:26.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00041112800000000003 2023-10-20T03:35:26.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.356 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.356 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.329703+0000", 2023-10-20T03:35:26.356 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8275999999999999e-05 2023-10-20T03:35:26.356 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.356 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.357 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.357 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.329715+0000", 2023-10-20T03:35:26.357 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2249000000000001e-05 2023-10-20T03:35:26.357 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.357 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.357 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.358 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.329788+0000", 2023-10-20T03:35:26.358 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2847000000000002e-05 2023-10-20T03:35:26.358 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.358 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.358 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.359 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.329797+0000", 2023-10-20T03:35:26.359 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0140000000000004e-06 2023-10-20T03:35:26.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.359 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.359 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.360 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.334204+0000", 2023-10-20T03:35:26.360 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044071099999999997 2023-10-20T03:35:26.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.360 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.361 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.334262+0000", 2023-10-20T03:35:26.361 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7682000000000001e-05 2023-10-20T03:35:26.361 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.361 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.361 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.362 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.362 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.1s1 43/31 ECSubWrite(tid=81, reqid=client.4570.0:307, at_version=43'15, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:26.362 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.342811+0000", 2023-10-20T03:35:26.362 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.85392953900000002, 2023-10-20T03:35:26.362 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056925029999999998, 2023-10-20T03:35:26.363 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.363 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.363 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.363 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.363 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.364 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.342811+0000", 2023-10-20T03:35:26.364 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.364 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.364 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.364 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.364 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.342811+0000", 2023-10-20T03:35:26.365 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.365 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.365 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.342812+0000", 2023-10-20T03:35:26.366 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4610000000000001e-06 2023-10-20T03:35:26.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.366 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.366 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.343155+0000", 2023-10-20T03:35:26.367 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00034283600000000001 2023-10-20T03:35:26.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.367 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.367 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.343176+0000", 2023-10-20T03:35:26.368 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1123000000000001e-05 2023-10-20T03:35:26.368 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.368 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.368 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.368 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.343188+0000", 2023-10-20T03:35:26.368 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2228e-05 2023-10-20T03:35:26.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.369 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.369 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.369 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.343273+0000", 2023-10-20T03:35:26.369 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4909999999999998e-05 2023-10-20T03:35:26.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.370 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.370 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.343287+0000", 2023-10-20T03:35:26.370 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3682e-05 2023-10-20T03:35:26.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.371 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.371 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.348463+0000", 2023-10-20T03:35:26.371 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0051761760000000002 2023-10-20T03:35:26.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.372 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.372 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.348503+0000", 2023-10-20T03:35:26.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0086999999999998e-05 2023-10-20T03:35:26.373 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.373 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.373 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.374 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.374 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.1s1 43/31 ECSubWrite(tid=83, reqid=client.4570.0:308, at_version=43'16, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:26.374 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.355748+0000", 2023-10-20T03:35:26.374 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.84099225200000005, 2023-10-20T03:35:26.374 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024802829999999998, 2023-10-20T03:35:26.374 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.375 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.375 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.375 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.375 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.375 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.355748+0000", 2023-10-20T03:35:26.376 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.376 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.376 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.355748+0000", 2023-10-20T03:35:26.377 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.377 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.377 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.355750+0000", 2023-10-20T03:35:26.378 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6270000000000001e-06 2023-10-20T03:35:26.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.355794+0000", 2023-10-20T03:35:26.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4252000000000003e-05 2023-10-20T03:35:26.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.379 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.379 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.355819+0000", 2023-10-20T03:35:26.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5289000000000001e-05 2023-10-20T03:35:26.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.380 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.380 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.355832+0000", 2023-10-20T03:35:26.380 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3139000000000001e-05 2023-10-20T03:35:26.381 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.381 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.381 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.381 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.355938+0000", 2023-10-20T03:35:26.381 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000105463 2023-10-20T03:35:26.381 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.382 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.382 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.382 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.355953+0000", 2023-10-20T03:35:26.382 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4875000000000001e-05 2023-10-20T03:35:26.382 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.383 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.383 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.383 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.358164+0000", 2023-10-20T03:35:26.383 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022118099999999998 2023-10-20T03:35:26.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.383 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.384 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.384 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.358228+0000", 2023-10-20T03:35:26.384 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3827999999999999e-05 2023-10-20T03:35:26.384 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.384 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.385 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.385 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.385 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.385 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.1s1 43/31 ECSubWrite(tid=84, reqid=client.4570.0:309, at_version=43'17, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:26.385 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.359496+0000", 2023-10-20T03:35:26.385 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.83724421400000004, 2023-10-20T03:35:26.386 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001733816, 2023-10-20T03:35:26.386 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.386 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.386 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.386 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.387 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.387 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.359496+0000", 2023-10-20T03:35:26.387 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.388 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.388 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.359496+0000", 2023-10-20T03:35:26.388 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.388 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.388 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.389 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.389 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.359497+0000", 2023-10-20T03:35:26.389 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1310000000000001e-06 2023-10-20T03:35:26.389 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.389 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.390 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.390 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.359521+0000", 2023-10-20T03:35:26.390 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3734e-05 2023-10-20T03:35:26.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.390 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.391 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.359545+0000", 2023-10-20T03:35:26.391 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3954000000000001e-05 2023-10-20T03:35:26.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.392 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.392 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.392 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.359567+0000", 2023-10-20T03:35:26.392 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2293000000000002e-05 2023-10-20T03:35:26.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.393 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.393 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.393 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.359647+0000", 2023-10-20T03:35:26.393 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9847999999999999e-05 2023-10-20T03:35:26.393 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.394 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.394 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.394 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.359660+0000", 2023-10-20T03:35:26.394 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3458000000000001e-05 2023-10-20T03:35:26.394 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.395 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.395 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.395 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.361171+0000", 2023-10-20T03:35:26.395 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001510869 2023-10-20T03:35:26.395 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.396 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.396 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.396 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.361230+0000", 2023-10-20T03:35:26.396 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8529000000000002e-05 2023-10-20T03:35:26.396 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.397 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.397 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.397 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.397 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.1s1 43/31 ECSubWrite(tid=86, reqid=client.4570.0:309, at_version=0'0, trim_to=0'0, roll_forward_to=43'17))", 2023-10-20T03:35:26.397 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.375239+0000", 2023-10-20T03:35:26.398 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.82150122999999997, 2023-10-20T03:35:26.398 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001093782, 2023-10-20T03:35:26.398 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.398 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.398 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.399 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.399 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.399 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.375239+0000", 2023-10-20T03:35:26.399 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.399 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.400 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.400 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.400 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.375239+0000", 2023-10-20T03:35:26.400 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.400 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.400 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.401 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.401 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.375240+0000", 2023-10-20T03:35:26.401 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4580000000000001e-06 2023-10-20T03:35:26.401 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.401 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.402 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.402 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.375256+0000", 2023-10-20T03:35:26.402 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5048999999999999e-05 2023-10-20T03:35:26.402 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.402 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.402 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.403 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.375276+0000", 2023-10-20T03:35:26.403 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0183e-05 2023-10-20T03:35:26.403 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.403 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.403 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.404 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.375287+0000", 2023-10-20T03:35:26.404 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1646e-05 2023-10-20T03:35:26.404 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.404 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.404 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.404 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.375380+0000", 2023-10-20T03:35:26.405 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2986999999999996e-05 2023-10-20T03:35:26.405 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.405 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.405 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.405 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.375395+0000", 2023-10-20T03:35:26.406 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4403000000000001e-05 2023-10-20T03:35:26.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.406 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.406 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.406 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.376279+0000", 2023-10-20T03:35:26.407 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00088429199999999996 2023-10-20T03:35:26.407 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.407 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.407 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.408 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.376333+0000", 2023-10-20T03:35:26.408 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3764000000000002e-05 2023-10-20T03:35:26.408 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.408 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.408 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.409 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.409 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpRead(3.1s1 43/31 ECSubRead(tid=87, to_read={3:8d0c2319:::smithi17039300-31:head=0,4096,0}, subchunks={3:8d0c2319:::smithi17039300-31:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:26.409 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.378269+0000", 2023-10-20T03:35:26.409 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.81847099599999995, 2023-10-20T03:35:26.409 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00027112600000000001, 2023-10-20T03:35:26.410 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.410 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.410 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.410 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.410 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.411 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.378269+0000", 2023-10-20T03:35:26.411 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.411 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.411 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.412 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.378269+0000", 2023-10-20T03:35:26.412 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.412 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.412 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.412 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.412 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.378270+0000", 2023-10-20T03:35:26.413 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8199999999999999e-07 2023-10-20T03:35:26.413 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.413 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.413 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.413 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.378280+0000", 2023-10-20T03:35:26.414 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8649999999999996e-06 2023-10-20T03:35:26.414 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.414 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.414 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.414 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.378296+0000", 2023-10-20T03:35:26.415 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5763000000000001e-05 2023-10-20T03:35:26.415 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.415 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.415 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.415 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.378308+0000", 2023-10-20T03:35:26.415 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2599000000000001e-05 2023-10-20T03:35:26.416 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.416 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.416 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.416 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.378390+0000", 2023-10-20T03:35:26.416 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1357e-05 2023-10-20T03:35:26.417 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.417 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.417 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.417 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.378540+0000", 2023-10-20T03:35:26.417 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015076 2023-10-20T03:35:26.417 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.418 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.418 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.418 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.418 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.418 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:26.421 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-10-20T03:35:26.422 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2023-10-20T03:35:26.425 INFO:tasks.rados.rados.0.smithi170.stdout:121: done (3 left) 2023-10-20T03:35:26.425 INFO:tasks.rados.rados.0.smithi170.stdout:122: done (2 left) 2023-10-20T03:35:26.425 INFO:tasks.rados.rados.0.smithi170.stdout:123: done (1 left) 2023-10-20T03:35:26.425 INFO:tasks.rados.rados.0.smithi170.stdout:125: done (0 left) 2023-10-20T03:35:26.426 INFO:tasks.rados.rados.0.smithi170.stdout:126: setattr oid 42 current snap is 8 2023-10-20T03:35:26.426 INFO:tasks.rados.rados.0.smithi170.stdout:127: rollback oid 18 current snap is 8 2023-10-20T03:35:26.426 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 18 to 6 2023-10-20T03:35:26.426 INFO:tasks.rados.rados.0.smithi170.stdout:128: setattr oid 33 current snap is 8 2023-10-20T03:35:26.426 INFO:tasks.rados.rados.0.smithi170.stdout:127: finishing rollback tid 1 to smithi17039300-18 2023-10-20T03:35:26.427 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 18 v 15 (ObjNum 289 snap 0 seq_num 0) dirty dne 2023-10-20T03:35:26.427 INFO:tasks.rados.rados.0.smithi170.stdout:127: done (2 left) 2023-10-20T03:35:26.427 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 49 current snap is 8 2023-10-20T03:35:26.427 INFO:tasks.rados.rados.0.smithi170.stdout:129: seq_num 80 ranges {819200=589824,1409024=573440,1982464=196608} 2023-10-20T03:35:26.427 INFO:tasks.rados.rados.0.smithi170.stdout:129: writing smithi17039300-49 from 819200 to 1409024 tid 1 2023-10-20T03:35:26.428 INFO:tasks.rados.rados.0.smithi170.stdout:129: writing smithi17039300-49 from 1409024 to 1982464 tid 2 2023-10-20T03:35:26.428 INFO:tasks.rados.rados.0.smithi170.stdout:129: writing smithi17039300-49 from 1982464 to 2179072 tid 3 2023-10-20T03:35:26.428 INFO:tasks.rados.rados.0.smithi170.stdout:130: rmattr oid 36 current snap is 8 2023-10-20T03:35:26.428 INFO:tasks.rados.rados.0.smithi170.stdout:130: done (3 left) 2023-10-20T03:35:26.428 INFO:tasks.rados.rados.0.smithi170.stdout:131: setattr oid 3 current snap is 8 2023-10-20T03:35:26.429 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 42 v 17 (ObjNum 60 snap 2 seq_num 60) dirty exists 2023-10-20T03:35:26.429 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 33 v 29 (ObjNum 52 snap 0 seq_num 52) dirty exists 2023-10-20T03:35:26.429 INFO:tasks.rados.rados.0.smithi170.stdout:126: done (3 left) 2023-10-20T03:35:26.429 INFO:tasks.rados.rados.0.smithi170.stdout:128: done (2 left) 2023-10-20T03:35:26.429 INFO:tasks.rados.rados.0.smithi170.stdout:132: rmattr oid 8 current snap is 8 2023-10-20T03:35:26.430 INFO:tasks.rados.rados.0.smithi170.stdout:132: done (2 left) 2023-10-20T03:35:26.430 INFO:tasks.rados.rados.0.smithi170.stdout:133: setattr oid 17 current snap is 8 2023-10-20T03:35:26.430 INFO:tasks.rados.rados.0.smithi170.stdout:134: rollback oid 35 current snap is 8 2023-10-20T03:35:26.430 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 35 to 6 2023-10-20T03:35:26.431 INFO:tasks.rados.rados.0.smithi170.stdout:135: rmattr oid 22 current snap is 8 2023-10-20T03:35:26.431 INFO:tasks.rados.rados.0.smithi170.stdout:135: done (4 left) 2023-10-20T03:35:26.431 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 34 current snap is 8 2023-10-20T03:35:26.431 INFO:tasks.rados.rados.0.smithi170.stdout:136: seq_num 81 ranges {557056=606208,1163264=720896,1884160=49152} 2023-10-20T03:35:26.431 INFO:tasks.rados.rados.0.smithi170.stdout:136: writing smithi17039300-34 from 557056 to 1163264 tid 1 2023-10-20T03:35:26.432 INFO:tasks.rados.rados.0.smithi170.stdout:136: writing smithi17039300-34 from 1163264 to 1884160 tid 2 2023-10-20T03:35:26.432 INFO:tasks.rados.rados.0.smithi170.stdout:136: writing smithi17039300-34 from 1884160 to 1933312 tid 3 2023-10-20T03:35:26.432 INFO:tasks.rados.rados.0.smithi170.stdout:137: copy_from oid 26 from oid 18 current snap is 8 2023-10-20T03:35:26.432 INFO:tasks.rados.rados.0.smithi170.stdout:134: finishing rollback tid 0 to smithi17039300-35 2023-10-20T03:35:26.432 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 3 v 31 (ObjNum 2 snap 0 seq_num 2) dirty exists 2023-10-20T03:35:26.433 INFO:tasks.rados.rados.0.smithi170.stdout:134: finishing rollback tid 1 to smithi17039300-35 2023-10-20T03:35:26.433 INFO:tasks.rados.rados.0.smithi170.stdout:134: finishing rollback tid 2 to smithi17039300-35 2023-10-20T03:35:26.433 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 23 (ObjNum 75 snap 4 seq_num 75) dirty exists 2023-10-20T03:35:26.433 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 33 (ObjNum 16 snap 0 seq_num 16) dirty exists 2023-10-20T03:35:26.433 INFO:tasks.rados.rados.0.smithi170.stdout:131: done (5 left) 2023-10-20T03:35:26.434 INFO:tasks.rados.rados.0.smithi170.stdout:133: done (4 left) 2023-10-20T03:35:26.434 INFO:tasks.rados.rados.0.smithi170.stdout:134: done (3 left) 2023-10-20T03:35:26.434 INFO:tasks.rados.rados.0.smithi170.stdout:138: setattr oid 29 current snap is 8 2023-10-20T03:35:26.434 INFO:tasks.rados.rados.0.smithi170.stdout:129: finishing write tid 1 to smithi17039300-49 2023-10-20T03:35:26.434 INFO:tasks.rados.rados.0.smithi170.stdout:139: delete oid 23 current snap is 8 2023-10-20T03:35:26.435 INFO:tasks.rados.rados.0.smithi170.stdout:137: finishing copy_from to smithi17039300-26 2023-10-20T03:35:26.435 INFO:tasks.rados.rados.0.smithi170.stdout:137: got expected ENOENT (src dne) 2023-10-20T03:35:26.435 INFO:tasks.rados.rados.0.smithi170.stdout:137: finishing copy_from racing read to smithi17039300-26 2023-10-20T03:35:26.435 INFO:tasks.rados.rados.0.smithi170.stdout:137: done (4 left) 2023-10-20T03:35:26.435 INFO:tasks.rados.rados.0.smithi170.stdout:139: done (3 left) 2023-10-20T03:35:26.436 INFO:tasks.rados.rados.0.smithi170.stdout:140: rmattr oid 17 current snap is 8 2023-10-20T03:35:26.436 INFO:tasks.rados.rados.0.smithi170.stdout:141: read oid 36 snap -1 2023-10-20T03:35:26.436 INFO:tasks.rados.rados.0.smithi170.stdout:141: expect (ObjNum 73 snap 4 seq_num 73) 2023-10-20T03:35:26.436 INFO:tasks.rados.rados.0.smithi170.stdout:142: snap_create 2023-10-20T03:35:26.436 INFO:tasks.rados.rados.0.smithi170.stdout:129: finishing write tid 2 to smithi17039300-49 2023-10-20T03:35:26.437 INFO:tasks.rados.rados.0.smithi170.stdout:129: finishing write tid 3 to smithi17039300-49 2023-10-20T03:35:26.437 INFO:tasks.rados.rados.0.smithi170.stdout:129: finishing write tid 4 to smithi17039300-49 2023-10-20T03:35:26.437 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 34 (ObjNum 16 snap 0 seq_num 16) dirty exists 2023-10-20T03:35:26.437 INFO:tasks.rados.rados.0.smithi170.stdout:129: finishing write tid 5 to smithi17039300-49 2023-10-20T03:35:26.437 INFO:tasks.rados.rados.0.smithi170.stdout:129: oid 49 updating version 0 to 10 2023-10-20T03:35:26.438 INFO:tasks.rados.rados.0.smithi170.stdout:129: oid 49 updating version 10 to 11 2023-10-20T03:35:26.438 INFO:tasks.rados.rados.0.smithi170.stdout:129: oid 49 updating version 11 to 12 2023-10-20T03:35:26.438 INFO:tasks.rados.rados.0.smithi170.stdout:129: oid 49 version 12 is already newer than 9 2023-10-20T03:35:26.438 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 49 v 12 (ObjNum 80 snap 8 seq_num 80) dirty exists 2023-10-20T03:35:26.438 INFO:tasks.rados.rados.0.smithi170.stdout:129: left oid 49 (ObjNum 80 snap 8 seq_num 80) 2023-10-20T03:35:26.439 INFO:tasks.rados.rados.0.smithi170.stdout:141: expect (ObjNum 73 snap 4 seq_num 73) 2023-10-20T03:35:26.439 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:26.417+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:26.448 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:35:26.494 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 29 v 12 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-10-20T03:35:26.495 INFO:tasks.rados.rados.0.smithi170.stdout:136: finishing write tid 1 to smithi17039300-34 2023-10-20T03:35:26.495 INFO:tasks.rados.rados.0.smithi170.stdout:136: finishing write tid 2 to smithi17039300-34 2023-10-20T03:35:26.495 INFO:tasks.rados.rados.0.smithi170.stdout:136: finishing write tid 3 to smithi17039300-34 2023-10-20T03:35:26.495 INFO:tasks.rados.rados.0.smithi170.stdout:136: finishing write tid 4 to smithi17039300-34 2023-10-20T03:35:26.496 INFO:tasks.rados.rados.0.smithi170.stdout:136: finishing write tid 5 to smithi17039300-34 2023-10-20T03:35:26.496 INFO:tasks.rados.rados.0.smithi170.stdout:136: oid 34 updating version 0 to 21 2023-10-20T03:35:26.496 INFO:tasks.rados.rados.0.smithi170.stdout:136: oid 34 version 21 is already newer than 19 2023-10-20T03:35:26.496 INFO:tasks.rados.rados.0.smithi170.stdout:136: oid 34 version 21 is already newer than 20 2023-10-20T03:35:26.496 INFO:tasks.rados.rados.0.smithi170.stdout:136: oid 34 version 21 is already newer than 18 2023-10-20T03:35:26.497 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 34 v 21 (ObjNum 81 snap 8 seq_num 81) dirty exists 2023-10-20T03:35:26.497 INFO:tasks.rados.rados.0.smithi170.stdout:136: left oid 34 (ObjNum 81 snap 8 seq_num 81) 2023-10-20T03:35:26.520 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:26.516+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:26.594 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:26.595 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:26.595 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:26.595 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:26.609 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-10-20T03:35:26.614 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:26.615 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:35:26.621 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:26.616+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:26.635 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:26.635 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:35:26.655 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:26.655 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:35:26.675 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:26.676 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:35:26.692 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:26.692 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:35:26.712 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:26.712 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:35:26.713 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:26.722 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:26.721+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:26.725 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:26.725 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:26.725 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:35:26.726 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:35:26.726 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:26.733 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:26.734 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:35:26.740 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-10-20T03:35:26.823 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:26.820+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:26.914 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:26.915 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:26.915 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:26.915 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:35:26.915 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.915 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=21, reqid=client.4570.0:198, at_version=36'13, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:26.916 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:26.916 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.6452413079999992, 2023-10-20T03:35:26.916 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086318609999999994, 2023-10-20T03:35:26.916 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.917 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.917 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.917 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.917 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.917 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:26.918 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.918 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.918 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:26.919 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.919 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.919 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:26.919 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.58e-07 2023-10-20T03:35:26.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.920 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.920 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.920 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.267902+0000", 2023-10-20T03:35:26.920 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014514409999999999 2023-10-20T03:35:26.921 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.921 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.921 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.921 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.267916+0000", 2023-10-20T03:35:26.921 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4664999999999999e-05 2023-10-20T03:35:26.921 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.922 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.922 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.267931+0000", 2023-10-20T03:35:26.922 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4192999999999999e-05 2023-10-20T03:35:26.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.923 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.923 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.923 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.268007+0000", 2023-10-20T03:35:26.923 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6058999999999998e-05 2023-10-20T03:35:26.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.924 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.924 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.268017+0000", 2023-10-20T03:35:26.924 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0648999999999999e-05 2023-10-20T03:35:26.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.925 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.925 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.925 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275053+0000", 2023-10-20T03:35:26.925 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0070355699999999997 2023-10-20T03:35:26.925 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.926 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.926 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275082+0000", 2023-10-20T03:35:26.926 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8926000000000001e-05 2023-10-20T03:35:26.926 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.927 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.927 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.927 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=22, reqid=client.4570.0:199, at_version=36'14, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:26.928 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.284006+0000", 2023-10-20T03:35:26.928 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.6276848610000005, 2023-10-20T03:35:26.928 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0061959429999999998, 2023-10-20T03:35:26.928 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.928 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.929 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.929 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.929 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.929 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284006+0000", 2023-10-20T03:35:26.929 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.930 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.930 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.930 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.930 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284006+0000", 2023-10-20T03:35:26.930 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.931 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.931 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.931 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284008+0000", 2023-10-20T03:35:26.931 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.398e-06 2023-10-20T03:35:26.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.932 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.932 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.932 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284090+0000", 2023-10-20T03:35:26.932 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2686000000000001e-05 2023-10-20T03:35:26.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.933 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.933 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.933 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284111+0000", 2023-10-20T03:35:26.933 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0363000000000001e-05 2023-10-20T03:35:26.933 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.934 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.934 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.934 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284123+0000", 2023-10-20T03:35:26.934 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2213e-05 2023-10-20T03:35:26.934 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.935 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.935 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.935 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284222+0000", 2023-10-20T03:35:26.935 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8587999999999999e-05 2023-10-20T03:35:26.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.936 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.936 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:26.936 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284237+0000", 2023-10-20T03:35:26.936 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.541e-05 2023-10-20T03:35:26.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.936 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.937 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:26.937 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.290159+0000", 2023-10-20T03:35:26.937 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0059216299999999998 2023-10-20T03:35:26.937 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.937 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.938 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.938 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.290202+0000", 2023-10-20T03:35:26.938 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3655000000000001e-05 2023-10-20T03:35:26.938 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.938 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.939 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.939 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:212 3.5s0 3:aed1dba0:::smithi17039300-29:2 [read 0~425984,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:26.939 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.277328+0000", 2023-10-20T03:35:26.940 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.6343637109999998, 2023-10-20T03:35:26.940 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085440599999999992, 2023-10-20T03:35:26.940 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.940 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:26.940 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:26.941 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:26.941 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:26.941 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 212 2023-10-20T03:35:26.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.941 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.942 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.942 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.942 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277328+0000", 2023-10-20T03:35:26.942 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.943 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.943 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.943 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277328+0000", 2023-10-20T03:35:26.943 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.944 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.944 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.944 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.944 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277330+0000", 2023-10-20T03:35:26.944 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7690000000000001e-06 2023-10-20T03:35:26.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.945 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.945 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.945 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277343+0000", 2023-10-20T03:35:26.945 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2221e-05 2023-10-20T03:35:26.946 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.946 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.946 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.946 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277368+0000", 2023-10-20T03:35:26.947 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5544000000000001e-05 2023-10-20T03:35:26.947 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.947 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.947 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.947 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277382+0000", 2023-10-20T03:35:26.948 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4246e-05 2023-10-20T03:35:26.948 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.948 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.948 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.948 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277510+0000", 2023-10-20T03:35:26.949 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012724599999999999 2023-10-20T03:35:26.949 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.949 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.949 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2023-10-20T03:35:26.949 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277524+0000", 2023-10-20T03:35:26.950 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4466e-05 2023-10-20T03:35:26.950 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.950 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.950 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.950 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.281643+0000", 2023-10-20T03:35:26.951 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041191680000000003 2023-10-20T03:35:26.951 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.951 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.951 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:26.952 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.281815+0000", 2023-10-20T03:35:26.952 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000171991 2023-10-20T03:35:26.952 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.952 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.952 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.953 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.285872+0000", 2023-10-20T03:35:26.953 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0040564090000000004 2023-10-20T03:35:26.953 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.953 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.953 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.954 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.954 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.954 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:213 3.5s0 3:aed1dba0:::smithi17039300-29:2 [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:26.954 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.277413+0000", 2023-10-20T03:35:26.954 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.6342778390000001, 2023-10-20T03:35:26.955 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0087379769999999992, 2023-10-20T03:35:26.955 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.955 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:26.955 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:26.955 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:26.956 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:26.956 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 213 2023-10-20T03:35:26.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.956 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.956 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.957 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.957 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277413+0000", 2023-10-20T03:35:26.957 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.957 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.957 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.958 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.958 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277413+0000", 2023-10-20T03:35:26.958 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.958 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.958 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.958 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.959 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277414+0000", 2023-10-20T03:35:26.959 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7599999999999997e-07 2023-10-20T03:35:26.959 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.959 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.959 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.960 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277418+0000", 2023-10-20T03:35:26.960 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3599999999999998e-06 2023-10-20T03:35:26.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.960 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.960 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.961 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277428+0000", 2023-10-20T03:35:26.961 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0034000000000001e-05 2023-10-20T03:35:26.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.961 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.961 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.962 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277435+0000", 2023-10-20T03:35:26.962 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2049999999999999e-06 2023-10-20T03:35:26.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.962 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.962 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.963 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277660+0000", 2023-10-20T03:35:26.963 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022565199999999999 2023-10-20T03:35:26.963 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.963 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.963 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:26.964 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277673+0000", 2023-10-20T03:35:26.964 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2330000000000001e-05 2023-10-20T03:35:26.964 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.964 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.964 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.965 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.282724+0000", 2023-10-20T03:35:26.965 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0050513370000000004 2023-10-20T03:35:26.965 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.965 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.965 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:26.966 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.282891+0000", 2023-10-20T03:35:26.966 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016650500000000001 2023-10-20T03:35:26.966 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.966 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.966 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.966 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.286151+0000", 2023-10-20T03:35:26.967 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003260878 2023-10-20T03:35:26.967 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.967 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.967 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.967 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.968 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.968 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:214 3.5s0 3:aed1dba0:::smithi17039300-29:2 [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:26.968 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.277483+0000", 2023-10-20T03:35:26.968 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.6342084510000001, 2023-10-20T03:35:26.968 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.009248025, 2023-10-20T03:35:26.969 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.969 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:26.969 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:26.969 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:26.969 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:26.969 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 214 2023-10-20T03:35:26.970 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.970 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.970 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.970 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.971 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277483+0000", 2023-10-20T03:35:26.971 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.971 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.971 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.971 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.972 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277483+0000", 2023-10-20T03:35:26.972 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.972 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.972 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.972 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.973 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277483+0000", 2023-10-20T03:35:26.973 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.75e-07 2023-10-20T03:35:26.973 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.973 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.973 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.974 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277489+0000", 2023-10-20T03:35:26.974 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.57e-06 2023-10-20T03:35:26.974 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.974 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.974 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.975 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277497+0000", 2023-10-20T03:35:26.975 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4419999999999998e-06 2023-10-20T03:35:26.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.975 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.975 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.976 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277504+0000", 2023-10-20T03:35:26.976 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3430000000000001e-06 2023-10-20T03:35:26.976 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.976 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.976 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.977 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277741+0000", 2023-10-20T03:35:26.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00023721699999999999 2023-10-20T03:35:26.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:26.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277751+0000", 2023-10-20T03:35:26.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0122999999999999e-05 2023-10-20T03:35:26.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.978 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.978 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.979 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283390+0000", 2023-10-20T03:35:26.979 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056394009999999996 2023-10-20T03:35:26.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.979 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:26.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283539+0000", 2023-10-20T03:35:26.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014805499999999999 2023-10-20T03:35:26.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.981 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.286731+0000", 2023-10-20T03:35:26.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003192299 2023-10-20T03:35:26.981 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.981 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.981 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.982 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.982 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpRead(3.5s0 37/31 ECSubRead(tid=11, to_read={3:aed1dba0:::smithi17039300-29:head=0,106496,0}, subchunks={3:aed1dba0:::smithi17039300-29:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:26.982 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.282540+0000", 2023-10-20T03:35:26.982 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.6291511869999997, 2023-10-20T03:35:26.983 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023131580000000001, 2023-10-20T03:35:26.983 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.983 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.983 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.983 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.984 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.282540+0000", 2023-10-20T03:35:26.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.984 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.282540+0000", 2023-10-20T03:35:26.985 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.985 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.986 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.986 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.282540+0000", 2023-10-20T03:35:26.986 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.986 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.987 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.987 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.282540+0000", 2023-10-20T03:35:26.987 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.987 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.987 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.988 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.282697+0000", 2023-10-20T03:35:26.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000156782 2023-10-20T03:35:26.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.988 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.989 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.282726+0000", 2023-10-20T03:35:26.989 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9042000000000001e-05 2023-10-20T03:35:26.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.989 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:26.990 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.284036+0000", 2023-10-20T03:35:26.990 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0013105129999999999 2023-10-20T03:35:26.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.990 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.990 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:26.991 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.284853+0000", 2023-10-20T03:35:26.991 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081682100000000002 2023-10-20T03:35:26.991 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.991 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:26.991 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:26.992 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.992 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.992 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpRead(3.5s0 37/31 ECSubRead(tid=12, to_read={3:aed1dba0:::smithi17039300-29:head=0,106496,0}, subchunks={3:aed1dba0:::smithi17039300-29:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:26.992 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.283242+0000", 2023-10-20T03:35:26.992 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.628448906, 2023-10-20T03:35:26.992 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0017236040000000001, 2023-10-20T03:35:26.993 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:26.993 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:26.993 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:26.993 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.993 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:26.994 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283242+0000", 2023-10-20T03:35:26.994 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.994 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.994 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:26.995 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283242+0000", 2023-10-20T03:35:26.995 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.995 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.995 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:26.996 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283242+0000", 2023-10-20T03:35:26.996 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.996 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.996 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:26.997 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283242+0000", 2023-10-20T03:35:26.997 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:26.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.997 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.997 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:26.998 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283288+0000", 2023-10-20T03:35:26.998 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5540000000000001e-05 2023-10-20T03:35:26.998 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.998 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.998 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:26.999 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283302+0000", 2023-10-20T03:35:26.999 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4503e-05 2023-10-20T03:35:26.999 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:26.999 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:26.999 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.000 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.284906+0000", 2023-10-20T03:35:27.000 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001603132 2023-10-20T03:35:27.000 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.000 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.001 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.001 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.284966+0000", 2023-10-20T03:35:27.001 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0429e-05 2023-10-20T03:35:27.001 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.001 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.002 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.002 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.002 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpReadReply(3.5s0 37/31 ECSubReadReply(tid=11, attrs_read=0))", 2023-10-20T03:35:27.002 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.283605+0000", 2023-10-20T03:35:27.003 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 7.6280860700000002, 2023-10-20T03:35:27.003 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001718507, 2023-10-20T03:35:27.003 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:27.003 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:27.003 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:27.004 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.004 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:27.004 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283605+0000", 2023-10-20T03:35:27.004 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.005 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.005 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:27.005 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283605+0000", 2023-10-20T03:35:27.005 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.005 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.006 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:27.006 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283606+0000", 2023-10-20T03:35:27.006 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4799999999999998e-07 2023-10-20T03:35:27.006 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.006 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.007 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:27.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.284113+0000", 2023-10-20T03:35:27.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00050717600000000002 2023-10-20T03:35:27.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.007 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:27.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.284121+0000", 2023-10-20T03:35:27.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8490000000000006e-06 2023-10-20T03:35:27.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.009 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:27.009 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.284125+0000", 2023-10-20T03:35:27.009 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2849999999999996e-06 2023-10-20T03:35:27.009 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.009 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.010 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.010 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.285286+0000", 2023-10-20T03:35:27.010 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011611799999999999 2023-10-20T03:35:27.010 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.010 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.011 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.011 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.285324+0000", 2023-10-20T03:35:27.011 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7469000000000003e-05 2023-10-20T03:35:27.011 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.011 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.011 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.012 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:316 3.5s0 3:aed1dba0:::smithi17039300-29:head [append 0~0] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:27.012 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.364850+0000", 2023-10-20T03:35:27.012 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 1.5468416710000001, 2023-10-20T03:35:27.013 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016187792999999999, 2023-10-20T03:35:27.013 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:27.013 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:27.013 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:27.013 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:27.014 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:27.014 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 316 2023-10-20T03:35:27.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.014 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:27.014 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.015 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:27.015 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364850+0000", 2023-10-20T03:35:27.015 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.015 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.015 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.016 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:27.016 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364850+0000", 2023-10-20T03:35:27.016 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.016 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.017 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:27.017 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364852+0000", 2023-10-20T03:35:27.017 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0260000000000002e-06 2023-10-20T03:35:27.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.017 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.018 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:27.018 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364861+0000", 2023-10-20T03:35:27.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5899999999999997e-06 2023-10-20T03:35:27.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.018 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.019 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:27.019 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364880+0000", 2023-10-20T03:35:27.019 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9245999999999998e-05 2023-10-20T03:35:27.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.019 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.019 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:27.020 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364892+0000", 2023-10-20T03:35:27.020 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.184e-05 2023-10-20T03:35:27.020 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.020 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.020 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.021 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364987+0000", 2023-10-20T03:35:27.021 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4752000000000004e-05 2023-10-20T03:35:27.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.021 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.021 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2023-10-20T03:35:27.022 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364997+0000", 2023-10-20T03:35:27.022 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0319999999999999e-05 2023-10-20T03:35:27.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.022 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.022 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.023 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.368108+0000", 2023-10-20T03:35:27.023 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0031111200000000002 2023-10-20T03:35:27.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.023 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.023 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:27.023 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.368196+0000", 2023-10-20T03:35:27.024 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7773000000000001e-05 2023-10-20T03:35:27.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.024 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.024 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:27.024 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.368815+0000", 2023-10-20T03:35:27.025 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00061905900000000004 2023-10-20T03:35:27.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.025 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.025 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:27.025 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370813+0000", 2023-10-20T03:35:27.026 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019981780000000002 2023-10-20T03:35:27.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.026 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.026 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:27.026 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.380943+0000", 2023-10-20T03:35:27.027 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010129487 2023-10-20T03:35:27.027 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.027 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.027 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.027 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.381037+0000", 2023-10-20T03:35:27.027 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4401999999999995e-05 2023-10-20T03:35:27.028 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.028 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.028 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.028 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.029 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:317 3.5s0 3:aed1dba0:::smithi17039300-29:head [rollback 9] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:27.029 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.364924+0000", 2023-10-20T03:35:27.029 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 1.5467668130000001, 2023-10-20T03:35:27.029 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017874489, 2023-10-20T03:35:27.029 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:27.030 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:27.030 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:27.030 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:27.030 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:27.030 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 317 2023-10-20T03:35:27.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.031 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:27.031 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.031 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:27.031 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364924+0000", 2023-10-20T03:35:27.031 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.032 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.032 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.032 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:27.032 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364924+0000", 2023-10-20T03:35:27.032 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.033 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.033 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.033 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:27.033 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364925+0000", 2023-10-20T03:35:27.033 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8599999999999998e-07 2023-10-20T03:35:27.033 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.034 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:27.034 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364929+0000", 2023-10-20T03:35:27.034 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7309999999999999e-06 2023-10-20T03:35:27.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.035 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:27.035 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364940+0000", 2023-10-20T03:35:27.035 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1236999999999999e-05 2023-10-20T03:35:27.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.036 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:27.036 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364947+0000", 2023-10-20T03:35:27.036 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0469999999999996e-06 2023-10-20T03:35:27.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.037 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.037 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365101+0000", 2023-10-20T03:35:27.037 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015391 2023-10-20T03:35:27.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.038 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:27.038 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365110+0000", 2023-10-20T03:35:27.038 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2e-06 2023-10-20T03:35:27.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.039 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.039 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.369303+0000", 2023-10-20T03:35:27.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041924149999999997 2023-10-20T03:35:27.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.040 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:27.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.369380+0000", 2023-10-20T03:35:27.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.716e-05 2023-10-20T03:35:27.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.041 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:27.041 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.369806+0000", 2023-10-20T03:35:27.041 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00042586099999999998 2023-10-20T03:35:27.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.042 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:27.042 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.372757+0000", 2023-10-20T03:35:27.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029512430000000001 2023-10-20T03:35:27.043 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.043 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.043 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:27.043 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.382729+0000", 2023-10-20T03:35:27.044 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099724180000000003 2023-10-20T03:35:27.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.044 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.044 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.044 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.382799+0000", 2023-10-20T03:35:27.045 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9680999999999999e-05 2023-10-20T03:35:27.045 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.045 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.045 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.046 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:318 3.5s0 3:aed1dba0:::smithi17039300-29:head [append 0~0] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:27.046 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:35:27.046 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 1.5467066599999999, 2023-10-20T03:35:27.046 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018045379, 2023-10-20T03:35:27.047 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:27.047 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:27.047 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:27.047 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:27.047 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:27.048 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 318 2023-10-20T03:35:27.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.048 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:27.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.048 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:27.049 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:35:27.049 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.049 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.049 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:27.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:35:27.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:27.051 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:35:27.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8899999999999999e-07 2023-10-20T03:35:27.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.051 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.052 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:27.052 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364989+0000", 2023-10-20T03:35:27.052 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7079999999999999e-06 2023-10-20T03:35:27.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.053 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:27.053 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364998+0000", 2023-10-20T03:35:27.053 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2229999999999996e-06 2023-10-20T03:35:27.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.054 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:27.054 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365004+0000", 2023-10-20T03:35:27.054 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1999999999999999e-06 2023-10-20T03:35:27.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365157+0000", 2023-10-20T03:35:27.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015222799999999999 2023-10-20T03:35:27.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:27.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365166+0000", 2023-10-20T03:35:27.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6739999999999992e-06 2023-10-20T03:35:27.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.057 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370187+0000", 2023-10-20T03:35:27.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0050205279999999998 2023-10-20T03:35:27.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:27.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370256+0000", 2023-10-20T03:35:27.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9151000000000003e-05 2023-10-20T03:35:27.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.059 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:27.059 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370618+0000", 2023-10-20T03:35:27.059 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00036223100000000002 2023-10-20T03:35:27.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.060 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:27.060 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.375514+0000", 2023-10-20T03:35:27.060 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0048961220000000001 2023-10-20T03:35:27.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.060 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:27.061 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.382954+0000", 2023-10-20T03:35:27.061 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074393280000000003 2023-10-20T03:35:27.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.061 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.383030+0000", 2023-10-20T03:35:27.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6396999999999994e-05 2023-10-20T03:35:27.062 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.062 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.063 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.063 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 44/31 ECSubWrite(tid=29, reqid=client.4570.0:320, at_version=44'17, trim_to=0'0, roll_forward_to=36'15))", 2023-10-20T03:35:27.063 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.342875+0000", 2023-10-20T03:35:27.064 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.56881668100000005, 2023-10-20T03:35:27.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042571989999999997, 2023-10-20T03:35:27.064 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:27.064 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:27.064 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:27.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.065 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:27.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.342875+0000", 2023-10-20T03:35:27.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.066 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:27.066 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.342875+0000", 2023-10-20T03:35:27.066 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:27.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.342875+0000", 2023-10-20T03:35:27.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.05e-07 2023-10-20T03:35:27.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.068 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:27.068 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.343268+0000", 2023-10-20T03:35:27.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000392941 2023-10-20T03:35:27.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:27.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.343280+0000", 2023-10-20T03:35:27.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2177e-05 2023-10-20T03:35:27.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:27.070 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.343289+0000", 2023-10-20T03:35:27.070 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1449999999999997e-06 2023-10-20T03:35:27.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.070 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.343395+0000", 2023-10-20T03:35:27.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000106277 2023-10-20T03:35:27.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:27.072 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.343406+0000", 2023-10-20T03:35:27.072 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0824e-05 2023-10-20T03:35:27.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.072 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:27.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.347041+0000", 2023-10-20T03:35:27.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0036352620000000002 2023-10-20T03:35:27.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.073 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.073 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.347132+0000", 2023-10-20T03:35:27.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0367999999999999e-05 2023-10-20T03:35:27.074 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.074 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.074 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.075 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:27.075 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:35:27.075 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.523025552, 2023-10-20T03:35:27.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028147078999999998, 2023-10-20T03:35:27.076 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:27.076 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:27.076 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:27.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:27.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:35:27.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:27.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:35:27.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.079 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:27.079 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:35:27.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4600000000000004e-07 2023-10-20T03:35:27.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:27.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388785+0000", 2023-10-20T03:35:27.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000118604 2023-10-20T03:35:27.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:27.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388827+0000", 2023-10-20T03:35:27.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2404000000000003e-05 2023-10-20T03:35:27.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:27.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388843+0000", 2023-10-20T03:35:27.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5894000000000001e-05 2023-10-20T03:35:27.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.389002+0000", 2023-10-20T03:35:27.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000158383 2023-10-20T03:35:27.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:27.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.389019+0000", 2023-10-20T03:35:27.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7266e-05 2023-10-20T03:35:27.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.085 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:27.085 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416738+0000", 2023-10-20T03:35:27.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027718779999999998 2023-10-20T03:35:27.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.085 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.086 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416813+0000", 2023-10-20T03:35:27.086 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5102e-05 2023-10-20T03:35:27.086 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.086 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.086 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.087 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:340 3.5s0 3:aed1dba0:::smithi17039300-29:head [create,setxattr nWX_d[W\\dXUWOgbXndrSbcTU`mgjR_]o\\nNcXpin (1403) in=40b,setxattr TSP[ddnm_WR]nh]QhSXpkpmmWeOP^b]baemT^Uqr (630) in=40b,setxattr R[QU^NWYrjfarmhlWmZlUkP\\ipWan`SnqZOX`VWh (806) in=40b,setxattr ]hV\\^[PQZO]n^abhilRfldoUhgtl\\\\oel`mgahd_ (3368) in=40b,setxattr OYc`XQ\\VjZfQ`]OsbemUo_mfVUSc^pNo\\[\\ldrhg (1087) in=40b,setxattr `pZtTmno_eqZOONfatgZ_`nnhemaaQWiteO]dc[V (175) in=40b,setxattr PdQa`feQapkrnjfOUYUeqsOt_hrsg\\noqrcV]pSh (981) in=40b,setxattr bk_QpaSsdlQfT`oa[pTQS_[ad]isfYm^tQNdhSiW (1718) in=40b,setxattr Ys`Y_UZ^PXhpgS`VS]`eVgkcmT^QW[aWscssRRSr (10033) in=40b,setxattr [jd]Q\\qemWTVcc`hiZnsrqrggetajZRa]\\_llS`h (1367) in=40b,setxattr `UajoeZh]tRSf\\[nnVat^TjN^_q`ooQ_br`itgZm (244) in=40b,setxattr Tng[WcWa_dsnRXtZNa^`fle`gQ___mar\\fgRsZVY (665) in=40b,setxattr nogmhepVZfQtnZc^cZSeaaogjklZkrff_Utg`UR^ (15856) in=40b,setxattr gfqSsQY[][rrac[gmaQrTSacNWttTTXhanmRa^fa (1998) in=40b,setxattr V`Y[ddb`_dfep[_]oUoTqQj\\UW[gkesnlgNPYtPa (1438) in=40b,setxattr YgoRVqUOj_mXPXsq^PdjaaolmQsTsTsZpaddktl[ (1400) in=40b,setxattr smn[QVPVkfVs]oTip]jtirOtU\\Pa\\Ym^poZgfZaq (1263) in=40b,setxattr TQSmSs^Z]\\rdosO_a^TOU^[p`jkQb`dXXh]s`Stq (909) in=40b,setxattr faPhpqVXmWh`VeNZo[b^[kYl^[[rYYPRrk_[ON`e (14664) in=40b,setxattr ksO_UmqqQefe[Q\\njt[W_roWeei`]Sn^TV]O^iSW (1963) in=40b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:27.087 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.397436+0000", 2023-10-20T03:35:27.087 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.51425485599999998, 2023-10-20T03:35:27.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024477860000000001, 2023-10-20T03:35:27.088 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:27.088 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:27.088 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:27.088 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:27.089 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:27.089 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 340 2023-10-20T03:35:27.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.089 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:27.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.090 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:27.090 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397436+0000", 2023-10-20T03:35:27.090 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.091 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.091 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:27.091 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397436+0000", 2023-10-20T03:35:27.091 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.092 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.092 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:27.092 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397439+0000", 2023-10-20T03:35:27.092 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6929999999999999e-06 2023-10-20T03:35:27.092 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.093 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:27.093 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397608+0000", 2023-10-20T03:35:27.093 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016918 2023-10-20T03:35:27.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.094 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.094 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:27.094 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397637+0000", 2023-10-20T03:35:27.094 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9057e-05 2023-10-20T03:35:27.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.095 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:27.095 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397653+0000", 2023-10-20T03:35:27.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5430999999999998e-05 2023-10-20T03:35:27.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.096 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.096 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.096 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397749+0000", 2023-10-20T03:35:27.096 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6694000000000006e-05 2023-10-20T03:35:27.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.097 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.097 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:27.097 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397877+0000", 2023-10-20T03:35:27.097 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012762599999999999 2023-10-20T03:35:27.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:27.098 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.399009+0000", 2023-10-20T03:35:27.098 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011323640000000001 2023-10-20T03:35:27.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.099 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.099 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:27.099 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416407+0000", 2023-10-20T03:35:27.099 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017398034999999999 2023-10-20T03:35:27.099 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.100 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:27.100 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.421802+0000", 2023-10-20T03:35:27.100 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053944320000000002 2023-10-20T03:35:27.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.101 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.101 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.101 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.421914+0000", 2023-10-20T03:35:27.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000112348 2023-10-20T03:35:27.101 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.102 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.102 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.102 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.102 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:27.103 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.401616+0000", 2023-10-20T03:35:27.103 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.51007510099999998, 2023-10-20T03:35:27.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015282946, 2023-10-20T03:35:27.103 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:27.103 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:27.104 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:27.104 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.104 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:27.104 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401616+0000", 2023-10-20T03:35:27.104 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.105 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:27.105 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401616+0000", 2023-10-20T03:35:27.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:27.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401617+0000", 2023-10-20T03:35:27.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.105e-06 2023-10-20T03:35:27.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.107 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.107 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:27.107 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401748+0000", 2023-10-20T03:35:27.107 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013050599999999999 2023-10-20T03:35:27.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.108 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.108 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:27.108 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401765+0000", 2023-10-20T03:35:27.108 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7450999999999999e-05 2023-10-20T03:35:27.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:27.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401774+0000", 2023-10-20T03:35:27.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9760000000000001e-06 2023-10-20T03:35:27.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.110 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.110 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.110 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404428+0000", 2023-10-20T03:35:27.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00265409 2023-10-20T03:35:27.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:27.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404442+0000", 2023-10-20T03:35:27.111 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3602e-05 2023-10-20T03:35:27.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.112 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:27.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416847+0000", 2023-10-20T03:35:27.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012404837 2023-10-20T03:35:27.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416899+0000", 2023-10-20T03:35:27.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2379000000000001e-05 2023-10-20T03:35:27.113 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.114 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.114 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.114 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=53, reqid=client.4570.0:335, at_version=44'20, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:27.115 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.403071+0000", 2023-10-20T03:35:27.115 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.50862010099999999, 2023-10-20T03:35:27.115 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014226947, 2023-10-20T03:35:27.115 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:27.115 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:27.115 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:27.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.116 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:27.116 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403071+0000", 2023-10-20T03:35:27.116 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.117 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.117 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:27.117 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403071+0000", 2023-10-20T03:35:27.117 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.117 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.117 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.118 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:27.118 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403072+0000", 2023-10-20T03:35:27.118 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2299999999999999e-07 2023-10-20T03:35:27.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.118 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.119 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:27.119 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403107+0000", 2023-10-20T03:35:27.119 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4900000000000001e-05 2023-10-20T03:35:27.119 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.119 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.120 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:27.120 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403125+0000", 2023-10-20T03:35:27.120 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8386999999999999e-05 2023-10-20T03:35:27.120 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.120 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.121 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:27.121 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403138+0000", 2023-10-20T03:35:27.121 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2979999999999999e-05 2023-10-20T03:35:27.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413715+0000", 2023-10-20T03:35:27.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010576538999999999 2023-10-20T03:35:27.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.123 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:27.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413733+0000", 2023-10-20T03:35:27.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8357000000000001e-05 2023-10-20T03:35:27.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:27.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417237+0000", 2023-10-20T03:35:27.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0035037689999999999 2023-10-20T03:35:27.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.124 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.125 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417298+0000", 2023-10-20T03:35:27.125 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1092000000000004e-05 2023-10-20T03:35:27.125 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.125 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.125 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.126 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.126 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 44/31 ECSubWriteReply(tid=22, last_complete=44'12, committed=1, applied=1))", 2023-10-20T03:35:27.126 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.404745+0000", 2023-10-20T03:35:27.126 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.50694664899999997, 2023-10-20T03:35:27.127 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089739039999999996, 2023-10-20T03:35:27.127 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:27.127 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:27.127 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:27.127 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.128 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:27.128 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404745+0000", 2023-10-20T03:35:27.128 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:27.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404745+0000", 2023-10-20T03:35:27.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.130 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:27.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404746+0000", 2023-10-20T03:35:27.130 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.156e-06 2023-10-20T03:35:27.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.131 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:27.131 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404753+0000", 2023-10-20T03:35:27.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7009999999999996e-06 2023-10-20T03:35:27.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.132 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:27.132 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404774+0000", 2023-10-20T03:35:27.132 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0390999999999998e-05 2023-10-20T03:35:27.132 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.132 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:27.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404790+0000", 2023-10-20T03:35:27.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5744e-05 2023-10-20T03:35:27.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413701+0000", 2023-10-20T03:35:27.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089117050000000007 2023-10-20T03:35:27.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413719+0000", 2023-10-20T03:35:27.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7207e-05 2023-10-20T03:35:27.135 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.135 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.135 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.136 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=54, reqid=client.4570.0:336, at_version=44'21, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:27.136 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.405096+0000", 2023-10-20T03:35:27.137 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.50659504399999999, 2023-10-20T03:35:27.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012879895000000001, 2023-10-20T03:35:27.137 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:27.137 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:27.137 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:27.138 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.138 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:27.138 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405096+0000", 2023-10-20T03:35:27.138 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.138 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.139 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:27.139 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405096+0000", 2023-10-20T03:35:27.139 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.140 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.140 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:27.140 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405097+0000", 2023-10-20T03:35:27.140 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.035e-06 2023-10-20T03:35:27.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.141 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.141 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:27.141 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405121+0000", 2023-10-20T03:35:27.141 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4097e-05 2023-10-20T03:35:27.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.142 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.142 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:27.142 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405177+0000", 2023-10-20T03:35:27.142 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5220000000000003e-05 2023-10-20T03:35:27.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.143 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.143 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:27.143 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405192+0000", 2023-10-20T03:35:27.143 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.579e-05 2023-10-20T03:35:27.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.144 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.144 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.144 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416447+0000", 2023-10-20T03:35:27.144 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011254188 2023-10-20T03:35:27.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.145 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.145 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:27.145 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416463+0000", 2023-10-20T03:35:27.145 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6441000000000001e-05 2023-10-20T03:35:27.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.146 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:27.146 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417913+0000", 2023-10-20T03:35:27.146 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014495809999999999 2023-10-20T03:35:27.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.147 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.147 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417976+0000", 2023-10-20T03:35:27.147 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3542999999999998e-05 2023-10-20T03:35:27.148 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.148 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.148 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.149 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 44/31 ECSubWriteReply(tid=22, last_complete=44'12, committed=1, applied=1))", 2023-10-20T03:35:27.149 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.406456+0000", 2023-10-20T03:35:27.149 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.505235196, 2023-10-20T03:35:27.149 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010387031999999999, 2023-10-20T03:35:27.149 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:27.150 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:27.150 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:27.150 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.150 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:27.150 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406456+0000", 2023-10-20T03:35:27.151 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.151 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:27.151 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406456+0000", 2023-10-20T03:35:27.152 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.152 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:27.152 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406457+0000", 2023-10-20T03:35:27.153 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6199999999999996e-07 2023-10-20T03:35:27.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.153 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.153 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:27.153 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406462+0000", 2023-10-20T03:35:27.154 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2889999999999997e-06 2023-10-20T03:35:27.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.154 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.154 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:27.154 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406479+0000", 2023-10-20T03:35:27.155 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6963e-05 2023-10-20T03:35:27.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.155 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.155 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:27.155 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406488+0000", 2023-10-20T03:35:27.155 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6200000000000005e-06 2023-10-20T03:35:27.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.156 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.156 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.156 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416476+0000", 2023-10-20T03:35:27.156 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099880660000000003 2023-10-20T03:35:27.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.157 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.157 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.157 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416843+0000", 2023-10-20T03:35:27.158 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000367232 2023-10-20T03:35:27.158 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.158 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.158 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.158 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.159 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 44/31 ECSubWriteReply(tid=22, last_complete=44'12, committed=1, applied=1))", 2023-10-20T03:35:27.159 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.406529+0000", 2023-10-20T03:35:27.159 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.50516194199999997, 2023-10-20T03:35:27.159 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010245637, 2023-10-20T03:35:27.160 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:27.160 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:27.160 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:27.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.160 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:27.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406529+0000", 2023-10-20T03:35:27.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.161 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:27.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406529+0000", 2023-10-20T03:35:27.162 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:27.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.162 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.162 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:27.162 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406530+0000", 2023-10-20T03:35:27.163 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9599999999999999e-07 2023-10-20T03:35:27.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.163 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:27.163 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406541+0000", 2023-10-20T03:35:27.164 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1208e-05 2023-10-20T03:35:27.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.164 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:27.164 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406557+0000", 2023-10-20T03:35:27.165 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5563999999999999e-05 2023-10-20T03:35:27.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.165 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:27.165 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406566+0000", 2023-10-20T03:35:27.165 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0769999999999997e-06 2023-10-20T03:35:27.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.166 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:27.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416751+0000", 2023-10-20T03:35:27.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010184794 2023-10-20T03:35:27.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:27.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:27.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:27.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416775+0000", 2023-10-20T03:35:27.167 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4298e-05 2023-10-20T03:35:27.168 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.168 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.168 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.168 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:27.168 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:27.169 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:27.170 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-10-20T03:35:27.171 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:27.171 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:35:27.172 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:27.172 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-10-20T03:35:27.172 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:27.173 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:35:27.173 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:27.174 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:35:27.174 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:27.175 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:35:27.175 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:27.176 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:35:27.176 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:35:27.069+0000 7f453ad9d640 0 -- 172.21.15.33:0/316625339 >> [v2:172.21.15.64:6816/2940007286,v1:172.21.15.64:6817/2940007286] conn(0x7f453c21c960 msgr2=0x7f453c21ed50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-10-20T03:35:27.176 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:27.168+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:27.177 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:27.177 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-10-20T03:35:27.177 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:27.178 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:27.179 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:35:27.179 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:27.271 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:27.269+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:27.337 INFO:tasks.rados.rados.0.smithi170.stdout:129: done (5 left) 2023-10-20T03:35:27.337 INFO:tasks.rados.rados.0.smithi170.stdout:136: done (4 left) 2023-10-20T03:35:27.337 INFO:tasks.rados.rados.0.smithi170.stdout:138: done (3 left) 2023-10-20T03:35:27.337 INFO:tasks.rados.rados.0.smithi170.stdout:140: done (2 left) 2023-10-20T03:35:27.338 INFO:tasks.rados.rados.0.smithi170.stdout:141: done (1 left) 2023-10-20T03:35:27.338 INFO:tasks.rados.rados.0.smithi170.stdout:142: done (0 left) 2023-10-20T03:35:27.338 INFO:tasks.rados.rados.0.smithi170.stdout:143: setattr oid 30 current snap is 9 2023-10-20T03:35:27.339 INFO:tasks.rados.rados.0.smithi170.stdout:144: read oid 38 snap -1 2023-10-20T03:35:27.339 INFO:tasks.rados.rados.0.smithi170.stdout:144: expect deleted 2023-10-20T03:35:27.339 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 49 current snap is 9 2023-10-20T03:35:27.339 INFO:tasks.rados.rados.0.smithi170.stdout:145: seq_num 83 ranges {2179072=638976,2818048=622592,3440640=147456} 2023-10-20T03:35:27.357 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:27.358 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:27.366 INFO:tasks.rados.rados.0.smithi170.stdout:145: writing smithi17039300-49 from 2179072 to 2818048 tid 1 2023-10-20T03:35:27.372 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-10-20T03:35:27.373 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:27.369+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:27.373 INFO:tasks.rados.rados.0.smithi170.stdout:145: writing smithi17039300-49 from 2818048 to 3440640 tid 2 2023-10-20T03:35:27.374 INFO:tasks.rados.rados.0.smithi170.stdout:145: writing smithi17039300-49 from 3440640 to 3588096 tid 3 2023-10-20T03:35:27.375 INFO:tasks.rados.rados.0.smithi170.stdout:146: delete oid 23 current snap is 9 2023-10-20T03:35:27.375 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 30 v 36 (ObjNum 29 snap 0 seq_num 29) dirty exists 2023-10-20T03:35:27.380 INFO:tasks.rados.rados.0.smithi170.stdout:143: done (3 left) 2023-10-20T03:35:27.380 INFO:tasks.rados.rados.0.smithi170.stdout:144: done (2 left) 2023-10-20T03:35:27.380 INFO:tasks.rados.rados.0.smithi170.stdout:146: done (1 left) 2023-10-20T03:35:27.381 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 37 current snap is 9 2023-10-20T03:35:27.381 INFO:tasks.rados.rados.0.smithi170.stdout:147: seq_num 85 ranges {1556480=671744,2228224=770048} 2023-10-20T03:35:27.402 INFO:tasks.rados.rados.0.smithi170.stdout:147: writing smithi17039300-37 from 1556480 to 2228224 tid 1 2023-10-20T03:35:27.411 INFO:tasks.rados.rados.0.smithi170.stdout:147: writing smithi17039300-37 from 2228224 to 2998272 tid 2 2023-10-20T03:35:27.411 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 14 current snap is 9 2023-10-20T03:35:27.412 INFO:tasks.rados.rados.0.smithi170.stdout:148: seq_num 86 ranges {1572864=688128,2260992=540672,2801664=229376} 2023-10-20T03:35:27.430 INFO:tasks.rados.rados.0.smithi170.stdout:148: writing smithi17039300-14 from 1572864 to 2260992 tid 1 2023-10-20T03:35:27.434 INFO:tasks.rados.rados.0.smithi170.stdout:148: writing smithi17039300-14 from 2260992 to 2801664 tid 2 2023-10-20T03:35:27.436 INFO:tasks.rados.rados.0.smithi170.stdout:148: writing smithi17039300-14 from 2801664 to 3031040 tid 3 2023-10-20T03:35:27.437 INFO:tasks.rados.rados.0.smithi170.stdout:149: rmattr oid 6 current snap is 9 2023-10-20T03:35:27.437 INFO:tasks.rados.rados.0.smithi170.stdout:149: done (3 left) 2023-10-20T03:35:27.437 INFO:tasks.rados.rados.0.smithi170.stdout:150: copy_from oid 13 from oid 30 current snap is 9 2023-10-20T03:35:27.437 INFO:tasks.rados.rados.0.smithi170.stdout:145: finishing write tid 1 to smithi17039300-49 2023-10-20T03:35:27.438 INFO:tasks.rados.rados.0.smithi170.stdout:151: snap_create 2023-10-20T03:35:27.438 INFO:tasks.rados.rados.0.smithi170.stdout:145: finishing write tid 2 to smithi17039300-49 2023-10-20T03:35:27.438 INFO:tasks.rados.rados.0.smithi170.stdout:145: finishing write tid 3 to smithi17039300-49 2023-10-20T03:35:27.438 INFO:tasks.rados.rados.0.smithi170.stdout:145: finishing write tid 4 to smithi17039300-49 2023-10-20T03:35:27.438 INFO:tasks.rados.rados.0.smithi170.stdout:145: finishing write tid 5 to smithi17039300-49 2023-10-20T03:35:27.439 INFO:tasks.rados.rados.0.smithi170.stdout:145: oid 49 updating version 0 to 16 2023-10-20T03:35:27.439 INFO:tasks.rados.rados.0.smithi170.stdout:145: oid 49 updating version 16 to 17 2023-10-20T03:35:27.439 INFO:tasks.rados.rados.0.smithi170.stdout:145: oid 49 version 17 is already newer than 15 2023-10-20T03:35:27.439 INFO:tasks.rados.rados.0.smithi170.stdout:145: oid 49 version 17 is already newer than 14 2023-10-20T03:35:27.439 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 49 v 17 (ObjNum 83 snap 9 seq_num 83) dirty exists 2023-10-20T03:35:27.440 INFO:tasks.rados.rados.0.smithi170.stdout:145: left oid 49 (ObjNum 83 snap 9 seq_num 83) 2023-10-20T03:35:27.441 INFO:tasks.rados.rados.0.smithi170.stdout:147: finishing write tid 1 to smithi17039300-37 2023-10-20T03:35:27.441 INFO:tasks.rados.rados.0.smithi170.stdout:147: finishing write tid 2 to smithi17039300-37 2023-10-20T03:35:27.442 INFO:tasks.rados.rados.0.smithi170.stdout:147: finishing write tid 3 to smithi17039300-37 2023-10-20T03:35:27.445 INFO:tasks.rados.rados.0.smithi170.stdout:147: finishing write tid 4 to smithi17039300-37 2023-10-20T03:35:27.445 INFO:tasks.rados.rados.0.smithi170.stdout:147: oid 37 updating version 0 to 18 2023-10-20T03:35:27.445 INFO:tasks.rados.rados.0.smithi170.stdout:147: oid 37 version 18 is already newer than 17 2023-10-20T03:35:27.445 INFO:tasks.rados.rados.0.smithi170.stdout:147: oid 37 updating version 18 to 19 2023-10-20T03:35:27.445 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 37 v 19 (ObjNum 85 snap 9 seq_num 85) dirty exists 2023-10-20T03:35:27.446 INFO:tasks.rados.rados.0.smithi170.stdout:147: left oid 37 (ObjNum 85 snap 9 seq_num 85) 2023-10-20T03:35:27.461 INFO:tasks.rados.rados.0.smithi170.stdout:148: finishing write tid 1 to smithi17039300-14 2023-10-20T03:35:27.467 INFO:tasks.rados.rados.0.smithi170.stdout:148: finishing write tid 2 to smithi17039300-14 2023-10-20T03:35:27.472 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:27.469+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:27.473 INFO:tasks.rados.rados.0.smithi170.stdout:148: finishing write tid 3 to smithi17039300-14 2023-10-20T03:35:27.476 INFO:tasks.rados.rados.0.smithi170.stdout:148: finishing write tid 4 to smithi17039300-14 2023-10-20T03:35:27.479 INFO:tasks.rados.rados.0.smithi170.stdout:148: finishing write tid 5 to smithi17039300-14 2023-10-20T03:35:27.479 INFO:tasks.rados.rados.0.smithi170.stdout:148: oid 14 updating version 0 to 38 2023-10-20T03:35:27.480 INFO:tasks.rados.rados.0.smithi170.stdout:148: oid 14 updating version 38 to 41 2023-10-20T03:35:27.480 INFO:tasks.rados.rados.0.smithi170.stdout:148: oid 14 version 41 is already newer than 40 2023-10-20T03:35:27.480 INFO:tasks.rados.rados.0.smithi170.stdout:148: oid 14 version 41 is already newer than 39 2023-10-20T03:35:27.480 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 41 (ObjNum 86 snap 9 seq_num 86) dirty exists 2023-10-20T03:35:27.480 INFO:tasks.rados.rados.0.smithi170.stdout:148: left oid 14 (ObjNum 86 snap 9 seq_num 86) 2023-10-20T03:35:27.483 INFO:tasks.rados.rados.0.smithi170.stdout:150: finishing copy_from racing read to smithi17039300-13 2023-10-20T03:35:27.496 INFO:tasks.rados.rados.0.smithi170.stdout:150: finishing copy_from to smithi17039300-13 2023-10-20T03:35:27.496 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 13 v 36 (ObjNum 29 snap 0 seq_num 29) dirty exists 2023-10-20T03:35:27.548 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:27.548 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:27.562 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-10-20T03:35:27.573 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:27.569+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:27.675 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:27.673+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:27.688 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:27.688 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:27.702 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-10-20T03:35:27.776 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:27.773+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:27.838 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:27.839 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:27.853 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-10-20T03:35:27.877 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:27.874+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:27.978 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:27.976+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:28.013 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:28.013 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:28.029 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-10-20T03:35:28.079 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:28.078+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:28.132 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [6] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 10, 14, 3, 7, 11, 15] 2023-10-20T03:35:28.133 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-10-20T03:35:28.133 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-10-20T03:35:28.133 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.073013 2023-10-20T03:35:28.134 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 0.07301267581133952 2023-10-20T03:35:28.154 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:28.155 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:28.169 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-10-20T03:35:28.182 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:28.181+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:28.280 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:28.277+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:28.297 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:28.297 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:28.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-10-20T03:35:28.343 INFO:tasks.rados.rados.0.smithi170.stdout:145: done (4 left) 2023-10-20T03:35:28.344 INFO:tasks.rados.rados.0.smithi170.stdout:147: done (3 left) 2023-10-20T03:35:28.344 INFO:tasks.rados.rados.0.smithi170.stdout:148: done (2 left) 2023-10-20T03:35:28.344 INFO:tasks.rados.rados.0.smithi170.stdout:150: done (1 left) 2023-10-20T03:35:28.344 INFO:tasks.rados.rados.0.smithi170.stdout:151: done (0 left) 2023-10-20T03:35:28.345 INFO:tasks.rados.rados.0.smithi170.stdout:152: snap_remove snap 8 2023-10-20T03:35:28.381 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:28.376+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:28.424 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-10-20T03:35:28.481 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:28.482 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:28.482 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:28.481+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:28.496 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-10-20T03:35:28.583 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:28.582+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:28.622 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:28.622 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:28.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-10-20T03:35:28.683 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:28.682+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:28.764 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:28.764 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:28.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-10-20T03:35:28.826 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:28.825+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:28.884 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:28.882+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:28.898 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:28.898 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:28.912 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-10-20T03:35:29.002 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:29.001+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:29.084 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:29.084 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:29.085 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:29.081+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:29.099 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-10-20T03:35:29.180 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:35:29.187 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:29.185+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:29.270 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:29.270 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:29.285 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-10-20T03:35:29.287 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:29.282+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:29.338 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:29.339 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.348 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2023-10-20T03:35:29.348 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.0 primary-affinity to 0.0730127 (847842) 2023-10-20T03:35:29.356 INFO:tasks.rados.rados.0.smithi170.stdout:152: done (0 left) 2023-10-20T03:35:29.356 INFO:tasks.rados.rados.0.smithi170.stdout:153: read oid 33 snap 5 2023-10-20T03:35:29.357 INFO:tasks.rados.rados.0.smithi170.stdout:153: expect (ObjNum 52 snap 0 seq_num 52) 2023-10-20T03:35:29.357 INFO:tasks.rados.rados.0.smithi170.stdout:154: copy_from oid 4 from oid 23 current snap is 10 2023-10-20T03:35:29.357 INFO:tasks.rados.rados.0.smithi170.stdout:155: rollback oid 46 current snap is 10 2023-10-20T03:35:29.357 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 46 to 3 2023-10-20T03:35:29.358 INFO:tasks.rados.rados.0.smithi170.stdout:156: copy_from oid 20 from oid 25 current snap is 10 2023-10-20T03:35:29.358 INFO:tasks.rados.rados.0.smithi170.stdout:157: rollback oid 21 current snap is 10 2023-10-20T03:35:29.358 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 21 to 0 2023-10-20T03:35:29.358 INFO:tasks.rados.rados.0.smithi170.stdout:158: snap_create 2023-10-20T03:35:29.359 INFO:tasks.rados.rados.0.smithi170.stdout:154: finishing copy_from to smithi17039300-4 2023-10-20T03:35:29.359 INFO:tasks.rados.rados.0.smithi170.stdout:154: got expected ENOENT (src dne) 2023-10-20T03:35:29.360 INFO:tasks.rados.rados.0.smithi170.stdout:154: finishing copy_from racing read to smithi17039300-4 2023-10-20T03:35:29.362 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:29.363 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.368 INFO:tasks.rados.rados.0.smithi170.stdout:153: expect (ObjNum 52 snap 0 seq_num 52) 2023-10-20T03:35:29.396 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:29.396 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.397 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:29.393+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:29.420 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:29.421 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.436 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:29.436 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.438 INFO:tasks.rados.rados.0.smithi170.stdout:157: finishing rollback tid 0 to smithi17039300-21 2023-10-20T03:35:29.438 INFO:tasks.rados.rados.0.smithi170.stdout:156: finishing copy_from racing read to smithi17039300-20 2023-10-20T03:35:29.438 INFO:tasks.rados.rados.0.smithi170.stdout:155: finishing rollback tid 0 to smithi17039300-46 2023-10-20T03:35:29.438 INFO:tasks.rados.rados.0.smithi170.stdout:157: finishing rollback tid 1 to smithi17039300-21 2023-10-20T03:35:29.439 INFO:tasks.rados.rados.0.smithi170.stdout:157: finishing rollback tid 2 to smithi17039300-21 2023-10-20T03:35:29.439 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 21 v 13 (ObjNum 20 snap 0 seq_num 20) dirty exists 2023-10-20T03:35:29.439 INFO:tasks.rados.rados.0.smithi170.stdout:155: finishing rollback tid 1 to smithi17039300-46 2023-10-20T03:35:29.439 INFO:tasks.rados.rados.0.smithi170.stdout:155: finishing rollback tid 2 to smithi17039300-46 2023-10-20T03:35:29.439 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 46 v 21 (ObjNum 45 snap 0 seq_num 45) dirty exists 2023-10-20T03:35:29.440 INFO:tasks.rados.rados.0.smithi170.stdout:156: finishing copy_from to smithi17039300-20 2023-10-20T03:35:29.440 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 20 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-10-20T03:35:29.456 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:29.457 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.457 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:29.463 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:29.463 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:29.477 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-10-20T03:35:29.478 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:29.479 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.497 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:29.497 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.516 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:29.517 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.522 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:29.521+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:29.539 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:29.540 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.562 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:29.562 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.581 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:29.582 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.589 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:29.585+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:29.603 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:29.604 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.605 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:29.605 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:35:29.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:35:29.606 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2023-10-20T03:35:29.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.606 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:306 3.1s0 3:8d0c2319:::smithi17039300-31:head [append 0~557056] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:29.606 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.325425+0000", 2023-10-20T03:35:29.606 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2779292260000004, 2023-10-20T03:35:29.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040045958999999999, 2023-10-20T03:35:29.607 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.607 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:29.607 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:29.607 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:29.608 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:29.608 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 306 2023-10-20T03:35:29.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.608 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.325425+0000", 2023-10-20T03:35:29.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.325425+0000", 2023-10-20T03:35:29.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.325427+0000", 2023-10-20T03:35:29.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.723e-06 2023-10-20T03:35:29.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.326339+0000", 2023-10-20T03:35:29.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00091144999999999996 2023-10-20T03:35:29.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.326373+0000", 2023-10-20T03:35:29.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4613000000000002e-05 2023-10-20T03:35:29.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.326391+0000", 2023-10-20T03:35:29.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8029e-05 2023-10-20T03:35:29.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.326524+0000", 2023-10-20T03:35:29.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013269900000000001 2023-10-20T03:35:29.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:29.616 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.326702+0000", 2023-10-20T03:35:29.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000178083 2023-10-20T03:35:29.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.328888+0000", 2023-10-20T03:35:29.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021861900000000002 2023-10-20T03:35:29.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.357500+0000", 2023-10-20T03:35:29.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.028611365 2023-10-20T03:35:29.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:29.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365350+0000", 2023-10-20T03:35:29.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0078497710000000002 2023-10-20T03:35:29.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365471+0000", 2023-10-20T03:35:29.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000121036 2023-10-20T03:35:29.620 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.620 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.620 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.621 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:307 3.1s0 3:8d0c2319:::smithi17039300-31:head [append 0~671744] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:29.621 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.337205+0000", 2023-10-20T03:35:29.621 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2661487520000003, 2023-10-20T03:35:29.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.028611688999999999, 2023-10-20T03:35:29.622 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.622 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:29.622 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:29.622 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:29.622 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:29.622 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 307 2023-10-20T03:35:29.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.623 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.337205+0000", 2023-10-20T03:35:29.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.337205+0000", 2023-10-20T03:35:29.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.337208+0000", 2023-10-20T03:35:29.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8040000000000002e-06 2023-10-20T03:35:29.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.338425+0000", 2023-10-20T03:35:29.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012170169999999999 2023-10-20T03:35:29.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.338469+0000", 2023-10-20T03:35:29.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4017000000000002e-05 2023-10-20T03:35:29.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.338492+0000", 2023-10-20T03:35:29.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2952999999999999e-05 2023-10-20T03:35:29.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.340429+0000", 2023-10-20T03:35:29.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019366419999999999 2023-10-20T03:35:29.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.630 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:29.630 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.340599+0000", 2023-10-20T03:35:29.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017065900000000001 2023-10-20T03:35:29.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.342200+0000", 2023-10-20T03:35:29.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0016006709999999999 2023-10-20T03:35:29.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.357762+0000", 2023-10-20T03:35:29.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015561896 2023-10-20T03:35:29.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:29.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365704+0000", 2023-10-20T03:35:29.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0079420069999999992 2023-10-20T03:35:29.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.634 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.634 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365817+0000", 2023-10-20T03:35:29.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011302300000000001 2023-10-20T03:35:29.634 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.635 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.635 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.636 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:308 3.1s0 3:8d0c2319:::smithi17039300-31:head [append 0~98304] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:29.636 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.338932+0000", 2023-10-20T03:35:29.636 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.264422218, 2023-10-20T03:35:29.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033689486999999997, 2023-10-20T03:35:29.636 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.637 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:29.637 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:29.637 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:29.637 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:29.637 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 308 2023-10-20T03:35:29.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.638 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.338932+0000", 2023-10-20T03:35:29.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.338932+0000", 2023-10-20T03:35:29.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.338934+0000", 2023-10-20T03:35:29.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0770000000000002e-06 2023-10-20T03:35:29.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339139+0000", 2023-10-20T03:35:29.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020497500000000001 2023-10-20T03:35:29.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339174+0000", 2023-10-20T03:35:29.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5583000000000001e-05 2023-10-20T03:35:29.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339194+0000", 2023-10-20T03:35:29.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9245999999999998e-05 2023-10-20T03:35:29.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.353975+0000", 2023-10-20T03:35:29.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014781949000000001 2023-10-20T03:35:29.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:29.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.354190+0000", 2023-10-20T03:35:29.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021416600000000001 2023-10-20T03:35:29.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.355201+0000", 2023-10-20T03:35:29.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0010116719999999999 2023-10-20T03:35:29.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.359820+0000", 2023-10-20T03:35:29.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0046182380000000002 2023-10-20T03:35:29.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:29.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.372433+0000", 2023-10-20T03:35:29.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012613509 2023-10-20T03:35:29.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.372621+0000", 2023-10-20T03:35:29.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018807200000000001 2023-10-20T03:35:29.650 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.650 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.650 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.651 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:309 3.1s0 3:8d0c2319:::smithi17039300-31:head [setxattr _header (56) in=7b] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:29.651 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.339226+0000", 2023-10-20T03:35:29.651 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2641283330000004, 2023-10-20T03:35:29.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035095548999999997, 2023-10-20T03:35:29.652 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.652 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:29.652 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:29.652 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:29.652 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:29.652 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 309 2023-10-20T03:35:29.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.653 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339226+0000", 2023-10-20T03:35:29.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339226+0000", 2023-10-20T03:35:29.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339227+0000", 2023-10-20T03:35:29.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.376e-06 2023-10-20T03:35:29.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339236+0000", 2023-10-20T03:35:29.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2799999999999992e-06 2023-10-20T03:35:29.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339256+0000", 2023-10-20T03:35:29.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9524000000000001e-05 2023-10-20T03:35:29.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339270+0000", 2023-10-20T03:35:29.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.47e-05 2023-10-20T03:35:29.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.357964+0000", 2023-10-20T03:35:29.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018693279 2023-10-20T03:35:29.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:29.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.358151+0000", 2023-10-20T03:35:29.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018687200000000001 2023-10-20T03:35:29.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.359027+0000", 2023-10-20T03:35:29.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087691600000000005 2023-10-20T03:35:29.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.361861+0000", 2023-10-20T03:35:29.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0028334319999999999 2023-10-20T03:35:29.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:29.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.374062+0000", 2023-10-20T03:35:29.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012201372 2023-10-20T03:35:29.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.374321+0000", 2023-10-20T03:35:29.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025879800000000002 2023-10-20T03:35:29.665 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.665 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.665 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.666 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:310 3.1s0 3:8d0c2319:::smithi17039300-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:29.666 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.339292+0000", 2023-10-20T03:35:29.666 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2640617540000001, 2023-10-20T03:35:29.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.042306519000000001, 2023-10-20T03:35:29.667 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.667 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:35:29.667 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:29.667 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:29.667 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:29.668 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 310 2023-10-20T03:35:29.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.668 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339292+0000", 2023-10-20T03:35:29.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339292+0000", 2023-10-20T03:35:29.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339294+0000", 2023-10-20T03:35:29.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4130000000000001e-06 2023-10-20T03:35:29.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339300+0000", 2023-10-20T03:35:29.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0499999999999997e-06 2023-10-20T03:35:29.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339334+0000", 2023-10-20T03:35:29.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4513e-05 2023-10-20T03:35:29.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339372+0000", 2023-10-20T03:35:29.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.769e-05 2023-10-20T03:35:29.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.359539+0000", 2023-10-20T03:35:29.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020167241999999998 2023-10-20T03:35:29.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:29.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.359694+0000", 2023-10-20T03:35:29.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000154585 2023-10-20T03:35:29.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.376927+0000", 2023-10-20T03:35:29.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017233637 2023-10-20T03:35:29.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:29.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.377208+0000", 2023-10-20T03:35:29.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028108099999999997 2023-10-20T03:35:29.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.381599+0000", 2023-10-20T03:35:29.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043903079999999999 2023-10-20T03:35:29.678 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.679 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.679 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:311 3.6s0 3:62cf25fd:::smithi17039300-14:head [append 0~573440] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:29.679 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.352581+0000", 2023-10-20T03:35:29.680 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.25077309, 2023-10-20T03:35:29.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032180940999999998, 2023-10-20T03:35:29.680 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.680 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:29.680 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:29.680 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:29.681 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:29.681 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 311 2023-10-20T03:35:29.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.681 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.352581+0000", 2023-10-20T03:35:29.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.352581+0000", 2023-10-20T03:35:29.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.352583+0000", 2023-10-20T03:35:29.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3099999999999999e-06 2023-10-20T03:35:29.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.353337+0000", 2023-10-20T03:35:29.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00075368099999999997 2023-10-20T03:35:29.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.353375+0000", 2023-10-20T03:35:29.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.854e-05 2023-10-20T03:35:29.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.353390+0000", 2023-10-20T03:35:29.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4538999999999999e-05 2023-10-20T03:35:29.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.353558+0000", 2023-10-20T03:35:29.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000168352 2023-10-20T03:35:29.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:29.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.353743+0000", 2023-10-20T03:35:29.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018488900000000001 2023-10-20T03:35:29.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.356274+0000", 2023-10-20T03:35:29.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025307340000000002 2023-10-20T03:35:29.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.384031+0000", 2023-10-20T03:35:29.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027757061 2023-10-20T03:35:29.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:29.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.384226+0000", 2023-10-20T03:35:29.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019552600000000001 2023-10-20T03:35:29.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.384762+0000", 2023-10-20T03:35:29.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053530899999999998 2023-10-20T03:35:29.693 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.693 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.693 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.694 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:312 3.6s0 3:62cf25fd:::smithi17039300-14:head [append 0~425984] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:29.694 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.360818+0000", 2023-10-20T03:35:29.694 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2425358129999999, 2023-10-20T03:35:29.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035316588000000003, 2023-10-20T03:35:29.695 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.695 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:29.695 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:29.695 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:29.695 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:29.696 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 312 2023-10-20T03:35:29.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.696 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.360818+0000", 2023-10-20T03:35:29.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.360818+0000", 2023-10-20T03:35:29.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.360820+0000", 2023-10-20T03:35:29.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1969999999999999e-06 2023-10-20T03:35:29.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.361437+0000", 2023-10-20T03:35:29.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00061634199999999997 2023-10-20T03:35:29.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.361474+0000", 2023-10-20T03:35:29.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7735999999999999e-05 2023-10-20T03:35:29.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.361497+0000", 2023-10-20T03:35:29.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2246999999999999e-05 2023-10-20T03:35:29.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369097+0000", 2023-10-20T03:35:29.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0076008430000000004 2023-10-20T03:35:29.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:29.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369369+0000", 2023-10-20T03:35:29.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000271721 2023-10-20T03:35:29.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.371397+0000", 2023-10-20T03:35:29.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0020282519999999999 2023-10-20T03:35:29.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.395428+0000", 2023-10-20T03:35:29.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024030335 2023-10-20T03:35:29.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:29.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.395694+0000", 2023-10-20T03:35:29.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026599799999999998 2023-10-20T03:35:29.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.396135+0000", 2023-10-20T03:35:29.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044091699999999998 2023-10-20T03:35:29.708 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.708 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.708 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.709 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:313 3.6s0 3:62cf25fd:::smithi17039300-14:head [append 0~344064] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:29.709 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.364439+0000", 2023-10-20T03:35:29.709 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2389149599999998, 2023-10-20T03:35:29.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032715017999999998, 2023-10-20T03:35:29.709 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.709 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:29.710 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:29.710 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:29.710 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:29.710 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 313 2023-10-20T03:35:29.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.711 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.364439+0000", 2023-10-20T03:35:29.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.364439+0000", 2023-10-20T03:35:29.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.364442+0000", 2023-10-20T03:35:29.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6129999999999998e-06 2023-10-20T03:35:29.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.364893+0000", 2023-10-20T03:35:29.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00045171099999999999 2023-10-20T03:35:29.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.364929+0000", 2023-10-20T03:35:29.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5366e-05 2023-10-20T03:35:29.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.364950+0000", 2023-10-20T03:35:29.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1399999999999998e-05 2023-10-20T03:35:29.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.385063+0000", 2023-10-20T03:35:29.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020113162 2023-10-20T03:35:29.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:29.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.385428+0000", 2023-10-20T03:35:29.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036503799999999998 2023-10-20T03:35:29.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.387009+0000", 2023-10-20T03:35:29.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0015803239999999999 2023-10-20T03:35:29.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.396145+0000", 2023-10-20T03:35:29.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0091364380000000002 2023-10-20T03:35:29.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.720 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:29.720 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.396872+0000", 2023-10-20T03:35:29.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072692500000000001 2023-10-20T03:35:29.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.397154+0000", 2023-10-20T03:35:29.721 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028204100000000002 2023-10-20T03:35:29.722 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.722 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.722 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.722 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:314 3.6s0 3:62cf25fd:::smithi17039300-14:head [setxattr _header (56) in=7b] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:29.723 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.364991+0000", 2023-10-20T03:35:29.723 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2383627209999997, 2023-10-20T03:35:29.723 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032796803999999999, 2023-10-20T03:35:29.723 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.723 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:29.724 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:29.724 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:29.724 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:29.724 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 314 2023-10-20T03:35:29.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.725 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.364991+0000", 2023-10-20T03:35:29.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.364991+0000", 2023-10-20T03:35:29.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.727 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.727 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.364993+0000", 2023-10-20T03:35:29.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5239999999999999e-06 2023-10-20T03:35:29.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365005+0000", 2023-10-20T03:35:29.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2033e-05 2023-10-20T03:35:29.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365030+0000", 2023-10-20T03:35:29.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5507000000000002e-05 2023-10-20T03:35:29.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365047+0000", 2023-10-20T03:35:29.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6753e-05 2023-10-20T03:35:29.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.393279+0000", 2023-10-20T03:35:29.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.028232090000000001 2023-10-20T03:35:29.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.732 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:29.732 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.393435+0000", 2023-10-20T03:35:29.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015599900000000001 2023-10-20T03:35:29.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.394369+0000", 2023-10-20T03:35:29.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00093351700000000003 2023-10-20T03:35:29.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.396532+0000", 2023-10-20T03:35:29.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021638640000000002 2023-10-20T03:35:29.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:29.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.397496+0000", 2023-10-20T03:35:29.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00096351599999999998 2023-10-20T03:35:29.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.397788+0000", 2023-10-20T03:35:29.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00029200100000000001 2023-10-20T03:35:29.736 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.737 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.737 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.737 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.737 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:315 3.6s0 3:62cf25fd:::smithi17039300-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:29.738 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.365076+0000", 2023-10-20T03:35:29.738 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2382783240000004, 2023-10-20T03:35:29.738 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.036384568999999999, 2023-10-20T03:35:29.738 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.739 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:35:29.739 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:29.739 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:29.739 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:29.739 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 315 2023-10-20T03:35:29.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.740 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.740 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.740 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365076+0000", 2023-10-20T03:35:29.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365076+0000", 2023-10-20T03:35:29.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.742 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.742 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365077+0000", 2023-10-20T03:35:29.742 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.426e-06 2023-10-20T03:35:29.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.743 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.743 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.743 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365084+0000", 2023-10-20T03:35:29.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2099999999999996e-06 2023-10-20T03:35:29.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.744 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.744 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.744 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365105+0000", 2023-10-20T03:35:29.744 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0633e-05 2023-10-20T03:35:29.744 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.744 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.745 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.745 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365119+0000", 2023-10-20T03:35:29.745 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.412e-05 2023-10-20T03:35:29.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.396242+0000", 2023-10-20T03:35:29.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031122606000000001 2023-10-20T03:35:29.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.747 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:29.747 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.396417+0000", 2023-10-20T03:35:29.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017512199999999999 2023-10-20T03:35:29.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.748 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.748 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.398627+0000", 2023-10-20T03:35:29.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022101170000000002 2023-10-20T03:35:29.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.748 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.748 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:29.749 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.398910+0000", 2023-10-20T03:35:29.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028353800000000001 2023-10-20T03:35:29.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.401460+0000", 2023-10-20T03:35:29.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025497969999999999 2023-10-20T03:35:29.750 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.750 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.750 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.751 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 43/31 ECSubWriteReply(tid=82, last_complete=43'24, committed=1, applied=1))", 2023-10-20T03:35:29.751 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.366655+0000", 2023-10-20T03:35:29.751 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2366986630000003, 2023-10-20T03:35:29.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016413984, 2023-10-20T03:35:29.752 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.752 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:29.752 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.752 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.753 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.366655+0000", 2023-10-20T03:35:29.753 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.366655+0000", 2023-10-20T03:35:29.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.755 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.755 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.366657+0000", 2023-10-20T03:35:29.755 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7570000000000001e-06 2023-10-20T03:35:29.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.756 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.756 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.366666+0000", 2023-10-20T03:35:29.756 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4069999999999999e-06 2023-10-20T03:35:29.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.756 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.757 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.757 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.366695+0000", 2023-10-20T03:35:29.757 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8989999999999999e-05 2023-10-20T03:35:29.757 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.757 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.757 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.758 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.366709+0000", 2023-10-20T03:35:29.758 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4079e-05 2023-10-20T03:35:29.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.758 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.758 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.759 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.383003+0000", 2023-10-20T03:35:29.759 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016293520999999998 2023-10-20T03:35:29.759 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.759 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.759 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.760 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.383069+0000", 2023-10-20T03:35:29.760 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6229999999999994e-05 2023-10-20T03:35:29.760 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.760 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.760 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.761 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.761 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.761 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 43/31 ECSubWriteReply(tid=82, last_complete=43'24, committed=1, applied=1))", 2023-10-20T03:35:29.761 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.366951+0000", 2023-10-20T03:35:29.761 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2364031549999996, 2023-10-20T03:35:29.761 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016235297999999999, 2023-10-20T03:35:29.762 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.762 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:29.762 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.762 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.762 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.366951+0000", 2023-10-20T03:35:29.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.765 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.766 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.766 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.366951+0000", 2023-10-20T03:35:29.766 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.766 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.767 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.767 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.366952+0000", 2023-10-20T03:35:29.767 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.767e-06 2023-10-20T03:35:29.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.767 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.768 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.366962+0000", 2023-10-20T03:35:29.768 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7569999999999999e-06 2023-10-20T03:35:29.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.769 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.769 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.366991+0000", 2023-10-20T03:35:29.769 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9149000000000002e-05 2023-10-20T03:35:29.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.769 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.770 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.770 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.367005+0000", 2023-10-20T03:35:29.770 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.366e-05 2023-10-20T03:35:29.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.770 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.771 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.771 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.383138+0000", 2023-10-20T03:35:29.771 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016133312 2023-10-20T03:35:29.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.771 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.772 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.772 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.383186+0000", 2023-10-20T03:35:29.772 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7652999999999999e-05 2023-10-20T03:35:29.772 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.772 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.773 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.773 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s1 43/31 ECSubWrite(tid=19, reqid=client.4570.0:317, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2023-10-20T03:35:29.773 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.370114+0000", 2023-10-20T03:35:29.774 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2332396150000005, 2023-10-20T03:35:29.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01192647, 2023-10-20T03:35:29.774 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.774 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:29.774 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370114+0000", 2023-10-20T03:35:29.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.776 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.776 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370114+0000", 2023-10-20T03:35:29.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370116+0000", 2023-10-20T03:35:29.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7230000000000001e-06 2023-10-20T03:35:29.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370167+0000", 2023-10-20T03:35:29.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0759e-05 2023-10-20T03:35:29.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370784+0000", 2023-10-20T03:35:29.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00061739900000000003 2023-10-20T03:35:29.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.780 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.780 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370810+0000", 2023-10-20T03:35:29.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5510000000000001e-05 2023-10-20T03:35:29.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.781 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.781 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.372961+0000", 2023-10-20T03:35:29.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021511030000000001 2023-10-20T03:35:29.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.782 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.782 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.372987+0000", 2023-10-20T03:35:29.782 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.658e-05 2023-10-20T03:35:29.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.783 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.783 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.381949+0000", 2023-10-20T03:35:29.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0089619899999999995 2023-10-20T03:35:29.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.784 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.784 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.382041+0000", 2023-10-20T03:35:29.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1405999999999999e-05 2023-10-20T03:35:29.784 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.785 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.785 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.785 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:322 3.6s0 3:6d58622e:::smithi17039300-33:head [create,setxattr c[X]j^UsOa`q]sYrSaa]raqNm]iSd^X]OmelObeN (242) in=40b,setxattr gopkVbealbs^Z^^`^sXjNlPPYp^\\fbk\\fUnmTq[R (208) in=40b,setxattr eUZoNnss (4560) in=8b,setxattr SYo`knsRbgPQPPbXW[[ZsTehoXbSPY_WQl]_URc` (105) in=40b,setxattr \\PYbW]PkbTb^sYhsihYgqj]pROo]\\NWqNhT_lWgn (17753) in=40b,setxattr ]R_fObdq\\NUZ]QiO]j`djkVVp`bfRdph]Zbea]gg (17332) in=40b,setxattr ]ptcajatsVQWlWeoNNlPngWealknZSiQarONqbaV (1473) in=40b,setxattr `i_qfTQWSto[gYWTmYsOtOhsW[fnhaVacqaadrU] (1191) in=40b,setxattr nNhpa\\e^VfUjte_Pc[kSor^[gpr[eaflXRs]SWZr (16140) in=40b,setxattr aRNZ_mZpadm^l[rZntn`UX^RN\\cZ_TT]rRTjibjc (1121) in=40b,setxattr aVipjRkr_^aolT[aa`UQhsfb\\eRQ[ZZleb_OQOlX (2384) in=40b,setxattr m^VcgrkhPRmZlXiSggOaoPTi^]^WnQh\\aXaekmoc (947) in=40b,setxattr NikdcN_ilNkWgOQgZqrYPT\\b[drPRq\\niPPQdbVU (5260) in=40b,setxattr ajnWlVTfP^PVRs[VchU^kT_gSmcjmZRh]ofVgTrp (1437) in=40b,setxattr liRPRT`aa[cqbZkpaWa\\q[NZWUXYk[a[cicigTsV (1017) in=40b,setxattr XkR_Nkkpdeiacga[]Y^NismNTd`tQP[eb[qbrrd^ (8912) in=40b,setxattr ]mai`f_WjqaYertRarRZ[mrmWf_`mktRW`slld^_ (243) in=40b,setxattr jacQPeR\\bNj_nOTOc^WkPrSpceTds[[eee`SndWY (1086) in=40b,setxattr \\lcQhm[Vm\\jW\\cPdSlWoZcU[d^htQTrhRak`t\\hk (4068) in=40b,setxattr e`Od\\\\V]jhaQVknTVdWZOTq[eYbmlehRf^ZtoZnQ (384) in=40b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:29.786 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.338383+0000", 2023-10-20T03:35:29.786 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.2649709960000002, 2023-10-20T03:35:29.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014835582, 2023-10-20T03:35:29.786 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.786 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:29.786 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:29.787 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:29.787 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:29.787 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 322 2023-10-20T03:35:29.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.787 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.338383+0000", 2023-10-20T03:35:29.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.338383+0000", 2023-10-20T03:35:29.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.338386+0000", 2023-10-20T03:35:29.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.123e-06 2023-10-20T03:35:29.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.790 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.338595+0000", 2023-10-20T03:35:29.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000209402 2023-10-20T03:35:29.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.338632+0000", 2023-10-20T03:35:29.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6473000000000001e-05 2023-10-20T03:35:29.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.792 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.338652+0000", 2023-10-20T03:35:29.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0169999999999998e-05 2023-10-20T03:35:29.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.793 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.793 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.338830+0000", 2023-10-20T03:35:29.793 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017752499999999999 2023-10-20T03:35:29.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.794 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op must wait for map", 2023-10-20T03:35:29.794 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.338843+0000", 2023-10-20T03:35:29.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3274e-05 2023-10-20T03:35:29.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.344137+0000", 2023-10-20T03:35:29.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005293776 2023-10-20T03:35:29.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:29.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.344349+0000", 2023-10-20T03:35:29.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021249499999999999 2023-10-20T03:35:29.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.797 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.797 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.346575+0000", 2023-10-20T03:35:29.797 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022262139999999998 2023-10-20T03:35:29.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.798 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.798 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.350058+0000", 2023-10-20T03:35:29.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0034831739999999999 2023-10-20T03:35:29.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.799 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:29.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.352921+0000", 2023-10-20T03:35:29.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002862191 2023-10-20T03:35:29.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.353218+0000", 2023-10-20T03:35:29.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00029776500000000002 2023-10-20T03:35:29.801 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.801 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.801 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.802 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:328 3.6s0 3:6e0199d0:::smithi17039300-3:head [create,setxattr ]iUi`W_blOlRsb\\cktWphl^jS`cU]lX`snt\\a``X (1893) in=40b,setxattr sOVifgTa[[YnXl\\pig\\ge_i\\bYohrcTZ\\tcNr\\rY (1473) in=40b,setxattr _O[^]bpfagp^`ZOOYZWP\\_fjNYbqhjbShkmttOrn (1963) in=40b,setxattr i\\ZtpdUPZ^tfd`Z\\mdhTeUsa`gpOpe^g\\ZX[^SdV (1050) in=40b,setxattr jRSlO`cdrcYdfsphScO`pVNUgWsNgtgiZaei[Vcg (593) in=40b,setxattr sqnegQOmPoe[\\`Rq\\O_Pke\\Up]qqa]sWdmSWOpna (314) in=40b,setxattr _akkVhR\\YsR`fma[XaYXUbm\\hPncrYZjTZ`_jZaV (35) in=40b,setxattr Qq^iU]_YXatV`eSUQ^pqcaao]o]YkVgUQahQcY`O (1471) in=40b,setxattr mQVRg^Ob[c`ZfnotUYbWh`XtjYPW\\Zdk[PeXnPmr (19508) in=40b,setxattr P`rlhmVNb]ggmVPfTXdYUoqnfNap]clhQXTTNSY] (384) in=40b,setxattr Ub^]aRWaR[pfhrO_`WYk[\\mST[krmXmc_VmccPoY (1578) in=40b,setxattr ZrRhdmep^[VQ]`m`UOcNi\\`VPbar[W]SdjVWYhrU (33) in=40b,setxattr sNn]UZbXjY^Oan]ct\\tof]VrSZXVRhrc]h`XOpZO (1295) in=40b,setxattr e\\[\\ZbkPZW^`Nf`_QacQqnTc`_`Vcntg^ck]Thgn (174) in=40b,setxattr bRkcddQRNclml^\\QVR[kObfggcgN]gsYOTl[YW]b (383) in=40b,setxattr i\\SZgUS^^W\\TabRhbk]baaokifbU\\Nr[sU]hXPdY (561) in=40b,setxattr OTnhiTm_UjdVqjcf[q[gPigcU\\aVafiakhoacOSr (841) in=40b,setxattr OcrWfpggPX_hZr[SZjisjjpl`ibkaehVPcfd]QOc (805) in=40b,setxattr oZjQQto]bVkNftdWSOsha]toecP_m^lhQYON`lfr (1541) in=40b,setxattr cmcl\\[l]NjZ__cNd[Tcmpa]Sibpb`\\qlSpb`idb^ (1998) in=40b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:29.802 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.368816+0000", 2023-10-20T03:35:29.802 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.2345379919999999, 2023-10-20T03:35:29.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011776249000000001, 2023-10-20T03:35:29.802 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.802 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:29.803 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:29.803 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:29.803 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:29.803 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 328 2023-10-20T03:35:29.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.804 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.804 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.804 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.368816+0000", 2023-10-20T03:35:29.804 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.805 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.368816+0000", 2023-10-20T03:35:29.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.806 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.806 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.368818+0000", 2023-10-20T03:35:29.806 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6369999999999999e-06 2023-10-20T03:35:29.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.807 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.368864+0000", 2023-10-20T03:35:29.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5442999999999998e-05 2023-10-20T03:35:29.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.368897+0000", 2023-10-20T03:35:29.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2957000000000003e-05 2023-10-20T03:35:29.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.368911+0000", 2023-10-20T03:35:29.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4323e-05 2023-10-20T03:35:29.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.810 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.810 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.810 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.369039+0000", 2023-10-20T03:35:29.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127892 2023-10-20T03:35:29.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:29.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.369239+0000", 2023-10-20T03:35:29.811 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019994500000000001 2023-10-20T03:35:29.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.370992+0000", 2023-10-20T03:35:29.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0017527140000000001 2023-10-20T03:35:29.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.378329+0000", 2023-10-20T03:35:29.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0073375710000000002 2023-10-20T03:35:29.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.814 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:29.814 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.379949+0000", 2023-10-20T03:35:29.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001619428 2023-10-20T03:35:29.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.815 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.815 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.380592+0000", 2023-10-20T03:35:29.815 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00064333900000000004 2023-10-20T03:35:29.815 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.815 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.816 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.816 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds5 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:29.816 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.388308+0000", 2023-10-20T03:35:29.817 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.2150455779999998, 2023-10-20T03:35:29.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.03103266, 2023-10-20T03:35:29.817 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.817 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:29.817 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.818 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.818 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388308+0000", 2023-10-20T03:35:29.818 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.819 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.819 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388308+0000", 2023-10-20T03:35:29.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388310+0000", 2023-10-20T03:35:29.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.823e-06 2023-10-20T03:35:29.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388817+0000", 2023-10-20T03:35:29.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00050723599999999999 2023-10-20T03:35:29.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388857+0000", 2023-10-20T03:35:29.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9387000000000001e-05 2023-10-20T03:35:29.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.823 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388880+0000", 2023-10-20T03:35:29.823 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3085999999999999e-05 2023-10-20T03:35:29.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.823 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.824 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.389046+0000", 2023-10-20T03:35:29.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000166439 2023-10-20T03:35:29.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.824 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.389075+0000", 2023-10-20T03:35:29.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8330000000000002e-05 2023-10-20T03:35:29.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419260+0000", 2023-10-20T03:35:29.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030185676000000002 2023-10-20T03:35:29.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419341+0000", 2023-10-20T03:35:29.827 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0683e-05 2023-10-20T03:35:29.827 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.827 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.827 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.828 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.828 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s1 44/31 ECSubWrite(tid=22, reqid=client.4570.0:340, at_version=44'12, trim_to=0'0, roll_forward_to=43'10))", 2023-10-20T03:35:29.828 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.399269+0000", 2023-10-20T03:35:29.828 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.2040851140000002, 2023-10-20T03:35:29.828 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020185995000000002, 2023-10-20T03:35:29.829 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.829 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:29.829 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.829 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.830 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399269+0000", 2023-10-20T03:35:29.830 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.830 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.831 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399269+0000", 2023-10-20T03:35:29.831 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.831 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.832 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399270+0000", 2023-10-20T03:35:29.832 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2999999999999999e-07 2023-10-20T03:35:29.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.832 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.832 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399458+0000", 2023-10-20T03:35:29.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018853500000000001 2023-10-20T03:35:29.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399484+0000", 2023-10-20T03:35:29.834 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5590000000000001e-05 2023-10-20T03:35:29.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.834 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399504+0000", 2023-10-20T03:35:29.835 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0667000000000001e-05 2023-10-20T03:35:29.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.835 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.835 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.400977+0000", 2023-10-20T03:35:29.835 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014726959999999999 2023-10-20T03:35:29.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.836 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.836 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401004+0000", 2023-10-20T03:35:29.836 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7336000000000001e-05 2023-10-20T03:35:29.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.837 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419375+0000", 2023-10-20T03:35:29.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018370537999999999 2023-10-20T03:35:29.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.838 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.838 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419455+0000", 2023-10-20T03:35:29.838 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9703000000000006e-05 2023-10-20T03:35:29.838 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.839 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.839 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.839 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds5 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:29.840 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.401536+0000", 2023-10-20T03:35:29.840 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.2018173679999999, 2023-10-20T03:35:29.840 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018405225000000001, 2023-10-20T03:35:29.840 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.840 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:29.841 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.841 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.841 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401536+0000", 2023-10-20T03:35:29.841 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401536+0000", 2023-10-20T03:35:29.842 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401538+0000", 2023-10-20T03:35:29.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7799999999999999e-06 2023-10-20T03:35:29.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.844 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.844 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402029+0000", 2023-10-20T03:35:29.844 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00049084400000000005 2023-10-20T03:35:29.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.845 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402063+0000", 2023-10-20T03:35:29.845 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3886e-05 2023-10-20T03:35:29.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.846 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.846 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402087+0000", 2023-10-20T03:35:29.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3932999999999999e-05 2023-10-20T03:35:29.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402235+0000", 2023-10-20T03:35:29.847 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014804900000000001 2023-10-20T03:35:29.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.848 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.848 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402262+0000", 2023-10-20T03:35:29.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7460000000000001e-05 2023-10-20T03:35:29.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.849 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.849 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419855+0000", 2023-10-20T03:35:29.849 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017593050999999998 2023-10-20T03:35:29.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419942+0000", 2023-10-20T03:35:29.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6222000000000002e-05 2023-10-20T03:35:29.850 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.851 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.851 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.851 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds5 44/31 ECSubWrite(tid=53, reqid=client.4570.0:335, at_version=44'20, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:29.852 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.404038+0000", 2023-10-20T03:35:29.852 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.1993159360000001, 2023-10-20T03:35:29.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016454106, 2023-10-20T03:35:29.852 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.852 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:29.853 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404038+0000", 2023-10-20T03:35:29.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404038+0000", 2023-10-20T03:35:29.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.855 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.855 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404039+0000", 2023-10-20T03:35:29.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.483e-06 2023-10-20T03:35:29.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404147+0000", 2023-10-20T03:35:29.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107556 2023-10-20T03:35:29.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.857 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404184+0000", 2023-10-20T03:35:29.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6683000000000001e-05 2023-10-20T03:35:29.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.858 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404207+0000", 2023-10-20T03:35:29.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3266000000000001e-05 2023-10-20T03:35:29.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.858 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.417674+0000", 2023-10-20T03:35:29.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013467478999999999 2023-10-20T03:35:29.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.417705+0000", 2023-10-20T03:35:29.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.112e-05 2023-10-20T03:35:29.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.860 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420397+0000", 2023-10-20T03:35:29.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002691756 2023-10-20T03:35:29.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420492+0000", 2023-10-20T03:35:29.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4763000000000001e-05 2023-10-20T03:35:29.862 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.862 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.862 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.863 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds5 44/31 ECSubWrite(tid=54, reqid=client.4570.0:336, at_version=44'21, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:29.863 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.405017+0000", 2023-10-20T03:35:29.863 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.1983365109999999, 2023-10-20T03:35:29.863 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016397957000000001, 2023-10-20T03:35:29.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:29.864 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:29.864 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:29.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:29.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.405017+0000", 2023-10-20T03:35:29.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:29.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.405017+0000", 2023-10-20T03:35:29.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:29.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:29.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.405019+0000", 2023-10-20T03:35:29.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1939999999999999e-06 2023-10-20T03:35:29.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:29.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.405055+0000", 2023-10-20T03:35:29.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6723000000000001e-05 2023-10-20T03:35:29.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:29.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.405085+0000", 2023-10-20T03:35:29.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9329999999999999e-05 2023-10-20T03:35:29.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:29.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.405109+0000", 2023-10-20T03:35:29.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4876e-05 2023-10-20T03:35:29.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:29.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419458+0000", 2023-10-20T03:35:29.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014348962 2023-10-20T03:35:29.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:29.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419482+0000", 2023-10-20T03:35:29.871 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3762000000000001e-05 2023-10-20T03:35:29.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.871 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.871 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:29.871 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421355+0000", 2023-10-20T03:35:29.872 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0018727069999999999 2023-10-20T03:35:29.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:29.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:29.872 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:29.873 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421415+0000", 2023-10-20T03:35:29.873 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0402999999999997e-05 2023-10-20T03:35:29.873 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.873 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.873 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.874 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:29.874 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:29.874 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:29.875 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:29.876 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:29.877 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-10-20T03:35:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:29.879 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:35:29.879 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:29.879 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:29.873+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:29.976 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:29.973+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:30.012 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2023-10-20T03:35:30.012 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:30.026 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-10-20T03:35:30.077 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:30.077+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:30.151 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2023-10-20T03:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2023-10-20T03:35:30.152 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:30.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-10-20T03:35:30.179 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:30.177+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:30.280 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:30.277+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:30.296 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:30.297 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2023-10-20T03:35:30.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2023-10-20T03:35:30.297 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2023-10-20T03:35:30.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.297 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s3 36/31 ECSubWrite(tid=60, reqid=client.4570.0:203, at_version=36'17, trim_to=0'0, roll_forward_to=34'14))", 2023-10-20T03:35:30.297 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:18.280060+0000", 2023-10-20T03:35:30.298 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.015537438999999, 2023-10-20T03:35:30.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042774780000000004, 2023-10-20T03:35:30.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.299 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280060+0000", 2023-10-20T03:35:30.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280060+0000", 2023-10-20T03:35:30.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280061+0000", 2023-10-20T03:35:30.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7099999999999999e-06 2023-10-20T03:35:30.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280118+0000", 2023-10-20T03:35:30.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6765000000000003e-05 2023-10-20T03:35:30.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280156+0000", 2023-10-20T03:35:30.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7679000000000003e-05 2023-10-20T03:35:30.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280182+0000", 2023-10-20T03:35:30.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6125000000000001e-05 2023-10-20T03:35:30.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280320+0000", 2023-10-20T03:35:30.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013794499999999999 2023-10-20T03:35:30.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280347+0000", 2023-10-20T03:35:30.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7489999999999999e-05 2023-10-20T03:35:30.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.284268+0000", 2023-10-20T03:35:30.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039205910000000002 2023-10-20T03:35:30.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.284337+0000", 2023-10-20T03:35:30.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9172999999999997e-05 2023-10-20T03:35:30.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.309 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.310 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s3 36/31 ECSubWrite(tid=61, reqid=client.4570.0:204, at_version=36'18, trim_to=0'0, roll_forward_to=34'14))", 2023-10-20T03:35:30.310 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:18.280230+0000", 2023-10-20T03:35:30.310 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.015367191999999, 2023-10-20T03:35:30.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004164002, 2023-10-20T03:35:30.311 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.311 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.311 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280230+0000", 2023-10-20T03:35:30.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280230+0000", 2023-10-20T03:35:30.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280231+0000", 2023-10-20T03:35:30.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3699999999999999e-07 2023-10-20T03:35:30.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280279+0000", 2023-10-20T03:35:30.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8149000000000003e-05 2023-10-20T03:35:30.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280305+0000", 2023-10-20T03:35:30.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6359e-05 2023-10-20T03:35:30.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.280345+0000", 2023-10-20T03:35:30.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9551999999999997e-05 2023-10-20T03:35:30.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.281038+0000", 2023-10-20T03:35:30.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069371700000000005 2023-10-20T03:35:30.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.281052+0000", 2023-10-20T03:35:30.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.332e-05 2023-10-20T03:35:30.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.284345+0000", 2023-10-20T03:35:30.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032929069999999999 2023-10-20T03:35:30.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.284394+0000", 2023-10-20T03:35:30.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9160999999999999e-05 2023-10-20T03:35:30.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.321 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.322 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.322 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs4 36/31 ECSubWrite(tid=32, reqid=client.4570.0:206, at_version=36'13, trim_to=0'0, roll_forward_to=34'11))", 2023-10-20T03:35:30.322 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:18.282036+0000", 2023-10-20T03:35:30.323 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.013561421, 2023-10-20T03:35:30.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068770749999999999, 2023-10-20T03:35:30.323 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.323 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.324 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.282036+0000", 2023-10-20T03:35:30.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.282036+0000", 2023-10-20T03:35:30.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.282036+0000", 2023-10-20T03:35:30.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9699999999999996e-07 2023-10-20T03:35:30.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.282087+0000", 2023-10-20T03:35:30.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0668999999999999e-05 2023-10-20T03:35:30.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.282120+0000", 2023-10-20T03:35:30.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2728999999999998e-05 2023-10-20T03:35:30.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.282143+0000", 2023-10-20T03:35:30.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3606000000000001e-05 2023-10-20T03:35:30.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.282375+0000", 2023-10-20T03:35:30.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000231318 2023-10-20T03:35:30.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.282470+0000", 2023-10-20T03:35:30.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5241000000000005e-05 2023-10-20T03:35:30.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.288810+0000", 2023-10-20T03:35:30.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0063400560000000002 2023-10-20T03:35:30.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.288913+0000", 2023-10-20T03:35:30.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102859 2023-10-20T03:35:30.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.334 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.335 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:30.335 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.329122+0000", 2023-10-20T03:35:30.335 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.966475086, 2023-10-20T03:35:30.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034738698999999998, 2023-10-20T03:35:30.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.335 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.336 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329122+0000", 2023-10-20T03:35:30.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329122+0000", 2023-10-20T03:35:30.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329122+0000", 2023-10-20T03:35:30.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6399999999999998e-07 2023-10-20T03:35:30.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329551+0000", 2023-10-20T03:35:30.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042853900000000002 2023-10-20T03:35:30.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329577+0000", 2023-10-20T03:35:30.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6075e-05 2023-10-20T03:35:30.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329594+0000", 2023-10-20T03:35:30.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7220000000000001e-05 2023-10-20T03:35:30.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329788+0000", 2023-10-20T03:35:30.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000193444 2023-10-20T03:35:30.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329815+0000", 2023-10-20T03:35:30.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7107e-05 2023-10-20T03:35:30.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.363772+0000", 2023-10-20T03:35:30.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033956919000000002 2023-10-20T03:35:30.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.363861+0000", 2023-10-20T03:35:30.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9030999999999995e-05 2023-10-20T03:35:30.345 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.345 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.346 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.346 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 43/31 ECSubWrite(tid=81, reqid=client.4570.0:307, at_version=43'15, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:30.346 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.342678+0000", 2023-10-20T03:35:30.347 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.952919004, 2023-10-20T03:35:30.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021268994999999999, 2023-10-20T03:35:30.347 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.347 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.347 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.342678+0000", 2023-10-20T03:35:30.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.342678+0000", 2023-10-20T03:35:30.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.342680+0000", 2023-10-20T03:35:30.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.736e-06 2023-10-20T03:35:30.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343038+0000", 2023-10-20T03:35:30.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035823799999999998 2023-10-20T03:35:30.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343066+0000", 2023-10-20T03:35:30.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7509e-05 2023-10-20T03:35:30.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343081+0000", 2023-10-20T03:35:30.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5676e-05 2023-10-20T03:35:30.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343490+0000", 2023-10-20T03:35:30.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040832599999999998 2023-10-20T03:35:30.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343517+0000", 2023-10-20T03:35:30.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7639000000000001e-05 2023-10-20T03:35:30.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.363880+0000", 2023-10-20T03:35:30.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020362392999999999 2023-10-20T03:35:30.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.363947+0000", 2023-10-20T03:35:30.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7477999999999993e-05 2023-10-20T03:35:30.357 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.357 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.357 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.358 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 43/31 ECSubWrite(tid=83, reqid=client.4570.0:308, at_version=43'16, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:30.358 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.355788+0000", 2023-10-20T03:35:30.358 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.9398093540000003, 2023-10-20T03:35:30.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0091204790000000008, 2023-10-20T03:35:30.359 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.359 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.359 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.355788+0000", 2023-10-20T03:35:30.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.355788+0000", 2023-10-20T03:35:30.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.355789+0000", 2023-10-20T03:35:30.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.59e-06 2023-10-20T03:35:30.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.355845+0000", 2023-10-20T03:35:30.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5241999999999997e-05 2023-10-20T03:35:30.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.355878+0000", 2023-10-20T03:35:30.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3776000000000003e-05 2023-10-20T03:35:30.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.355904+0000", 2023-10-20T03:35:30.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5655000000000001e-05 2023-10-20T03:35:30.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.361027+0000", 2023-10-20T03:35:30.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051232300000000003 2023-10-20T03:35:30.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.361063+0000", 2023-10-20T03:35:30.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6059000000000002e-05 2023-10-20T03:35:30.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.364822+0000", 2023-10-20T03:35:30.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037588360000000002 2023-10-20T03:35:30.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.364908+0000", 2023-10-20T03:35:30.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6091e-05 2023-10-20T03:35:30.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.370 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.370 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.370 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s3 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:35:30.371 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.357002+0000", 2023-10-20T03:35:30.371 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.9385952599999996, 2023-10-20T03:35:30.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024440087999999999, 2023-10-20T03:35:30.371 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.371 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.372 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357002+0000", 2023-10-20T03:35:30.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357002+0000", 2023-10-20T03:35:30.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357004+0000", 2023-10-20T03:35:30.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.883e-06 2023-10-20T03:35:30.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357470+0000", 2023-10-20T03:35:30.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046657099999999999 2023-10-20T03:35:30.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357514+0000", 2023-10-20T03:35:30.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3288999999999998e-05 2023-10-20T03:35:30.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357533+0000", 2023-10-20T03:35:30.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9123e-05 2023-10-20T03:35:30.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357687+0000", 2023-10-20T03:35:30.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000154389 2023-10-20T03:35:30.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357703+0000", 2023-10-20T03:35:30.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6062999999999999e-05 2023-10-20T03:35:30.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.381375+0000", 2023-10-20T03:35:30.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023672364000000001 2023-10-20T03:35:30.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.381442+0000", 2023-10-20T03:35:30.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6406000000000001e-05 2023-10-20T03:35:30.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.384 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.384 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.384 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 43/31 ECSubWrite(tid=84, reqid=client.4570.0:309, at_version=43'17, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:30.385 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.359441+0000", 2023-10-20T03:35:30.385 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.9361555749999999, 2023-10-20T03:35:30.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0062641490000000001, 2023-10-20T03:35:30.385 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.385 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.386 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359441+0000", 2023-10-20T03:35:30.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359441+0000", 2023-10-20T03:35:30.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359443+0000", 2023-10-20T03:35:30.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.689e-06 2023-10-20T03:35:30.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359487+0000", 2023-10-20T03:35:30.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4353e-05 2023-10-20T03:35:30.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359527+0000", 2023-10-20T03:35:30.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9394999999999998e-05 2023-10-20T03:35:30.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359551+0000", 2023-10-20T03:35:30.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3669999999999999e-05 2023-10-20T03:35:30.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.364039+0000", 2023-10-20T03:35:30.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044881959999999999 2023-10-20T03:35:30.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.364060+0000", 2023-10-20T03:35:30.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1588999999999999e-05 2023-10-20T03:35:30.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365612+0000", 2023-10-20T03:35:30.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001551569 2023-10-20T03:35:30.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365706+0000", 2023-10-20T03:35:30.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3688000000000002e-05 2023-10-20T03:35:30.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.396 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.397 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s3 43/31 ECSubWrite(tid=85, reqid=client.4570.0:312, at_version=43'25, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:35:30.397 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.372402+0000", 2023-10-20T03:35:30.397 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.9231946769999997, 2023-10-20T03:35:30.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097873730000000003, 2023-10-20T03:35:30.397 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.397 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.398 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.372402+0000", 2023-10-20T03:35:30.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.372402+0000", 2023-10-20T03:35:30.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.372404+0000", 2023-10-20T03:35:30.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.643e-06 2023-10-20T03:35:30.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.372682+0000", 2023-10-20T03:35:30.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027765899999999998 2023-10-20T03:35:30.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.372711+0000", 2023-10-20T03:35:30.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8872000000000001e-05 2023-10-20T03:35:30.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.372732+0000", 2023-10-20T03:35:30.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1455e-05 2023-10-20T03:35:30.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.374102+0000", 2023-10-20T03:35:30.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001369536 2023-10-20T03:35:30.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.374123+0000", 2023-10-20T03:35:30.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1939000000000001e-05 2023-10-20T03:35:30.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.382132+0000", 2023-10-20T03:35:30.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080084490000000008 2023-10-20T03:35:30.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.382190+0000", 2023-10-20T03:35:30.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7819999999999999e-05 2023-10-20T03:35:30.407 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.408 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s3 43/31 ECSubWrite(tid=88, reqid=client.4570.0:313, at_version=43'26, trim_to=0'0, roll_forward_to=43'24))", 2023-10-20T03:35:30.409 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.387663+0000", 2023-10-20T03:35:30.409 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.9079337059999997, 2023-10-20T03:35:30.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.006090944, 2023-10-20T03:35:30.409 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.409 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.410 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.387663+0000", 2023-10-20T03:35:30.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.387663+0000", 2023-10-20T03:35:30.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.387665+0000", 2023-10-20T03:35:30.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5260000000000001e-06 2023-10-20T03:35:30.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.387802+0000", 2023-10-20T03:35:30.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137556 2023-10-20T03:35:30.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.387900+0000", 2023-10-20T03:35:30.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7467999999999996e-05 2023-10-20T03:35:30.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.387938+0000", 2023-10-20T03:35:30.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8405999999999998e-05 2023-10-20T03:35:30.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.388054+0000", 2023-10-20T03:35:30.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115662 2023-10-20T03:35:30.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.388069+0000", 2023-10-20T03:35:30.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4589e-05 2023-10-20T03:35:30.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.393684+0000", 2023-10-20T03:35:30.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056154159999999998 2023-10-20T03:35:30.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.393754+0000", 2023-10-20T03:35:30.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0321000000000007e-05 2023-10-20T03:35:30.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.420 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.421 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s3 44/31 ECSubWrite(tid=92, reqid=client.4570.0:322, at_version=44'29, trim_to=0'0, roll_forward_to=43'27))", 2023-10-20T03:35:30.421 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.346792+0000", 2023-10-20T03:35:30.421 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.948804773, 2023-10-20T03:35:30.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0049682579999999997, 2023-10-20T03:35:30.421 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.422 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.422 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.346792+0000", 2023-10-20T03:35:30.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.346792+0000", 2023-10-20T03:35:30.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.346794+0000", 2023-10-20T03:35:30.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.316e-06 2023-10-20T03:35:30.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.347130+0000", 2023-10-20T03:35:30.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033612099999999999 2023-10-20T03:35:30.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.347161+0000", 2023-10-20T03:35:30.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1365999999999998e-05 2023-10-20T03:35:30.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.347185+0000", 2023-10-20T03:35:30.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4006e-05 2023-10-20T03:35:30.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.347366+0000", 2023-10-20T03:35:30.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000181118 2023-10-20T03:35:30.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.347392+0000", 2023-10-20T03:35:30.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5582e-05 2023-10-20T03:35:30.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.351654+0000", 2023-10-20T03:35:30.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042622559999999999 2023-10-20T03:35:30.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.351761+0000", 2023-10-20T03:35:30.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106493 2023-10-20T03:35:30.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.432 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.433 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/31 ECSubWrite(tid=18, reqid=client.4570.0:323, at_version=44'9, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:35:30.433 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.360382+0000", 2023-10-20T03:35:30.433 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9352145759999999, 2023-10-20T03:35:30.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012825586999999999, 2023-10-20T03:35:30.434 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.434 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.434 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.360382+0000", 2023-10-20T03:35:30.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.360382+0000", 2023-10-20T03:35:30.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.360383+0000", 2023-10-20T03:35:30.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7000000000000003e-07 2023-10-20T03:35:30.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.360923+0000", 2023-10-20T03:35:30.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053936100000000003 2023-10-20T03:35:30.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.360949+0000", 2023-10-20T03:35:30.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6593000000000001e-05 2023-10-20T03:35:30.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.360964+0000", 2023-10-20T03:35:30.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4499e-05 2023-10-20T03:35:30.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.361093+0000", 2023-10-20T03:35:30.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012922399999999999 2023-10-20T03:35:30.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.361111+0000", 2023-10-20T03:35:30.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7742000000000001e-05 2023-10-20T03:35:30.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.373129+0000", 2023-10-20T03:35:30.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012017724 2023-10-20T03:35:30.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.373208+0000", 2023-10-20T03:35:30.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9474000000000005e-05 2023-10-20T03:35:30.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.445 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.445 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.445 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/31 ECSubWrite(tid=19, reqid=client.4570.0:324, at_version=44'10, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:35:30.445 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.381061+0000", 2023-10-20T03:35:30.446 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9145357700000001, 2023-10-20T03:35:30.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011569289, 2023-10-20T03:35:30.446 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.446 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.446 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.381061+0000", 2023-10-20T03:35:30.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.381061+0000", 2023-10-20T03:35:30.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.381063+0000", 2023-10-20T03:35:30.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.48e-06 2023-10-20T03:35:30.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.381232+0000", 2023-10-20T03:35:30.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000169529 2023-10-20T03:35:30.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.381263+0000", 2023-10-20T03:35:30.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1006000000000001e-05 2023-10-20T03:35:30.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.381278+0000", 2023-10-20T03:35:30.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5136e-05 2023-10-20T03:35:30.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.381398+0000", 2023-10-20T03:35:30.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011959300000000001 2023-10-20T03:35:30.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.381413+0000", 2023-10-20T03:35:30.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4545999999999999e-05 2023-10-20T03:35:30.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.392546+0000", 2023-10-20T03:35:30.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011133412000000001 2023-10-20T03:35:30.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.392631+0000", 2023-10-20T03:35:30.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4586999999999995e-05 2023-10-20T03:35:30.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.457 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.458 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/31 ECSubWrite(tid=20, reqid=client.4570.0:325, at_version=44'11, trim_to=0'0, roll_forward_to=44'9))", 2023-10-20T03:35:30.458 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.398599+0000", 2023-10-20T03:35:30.458 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8969979860000001, 2023-10-20T03:35:30.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071836010000000004, 2023-10-20T03:35:30.459 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.459 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.459 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.398599+0000", 2023-10-20T03:35:30.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.398599+0000", 2023-10-20T03:35:30.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.398601+0000", 2023-10-20T03:35:30.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5400000000000001e-06 2023-10-20T03:35:30.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.398676+0000", 2023-10-20T03:35:30.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5204999999999996e-05 2023-10-20T03:35:30.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.398707+0000", 2023-10-20T03:35:30.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0902000000000003e-05 2023-10-20T03:35:30.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.398722+0000", 2023-10-20T03:35:30.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5543e-05 2023-10-20T03:35:30.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.398846+0000", 2023-10-20T03:35:30.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012341300000000001 2023-10-20T03:35:30.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.398863+0000", 2023-10-20T03:35:30.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7450000000000001e-05 2023-10-20T03:35:30.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.405717+0000", 2023-10-20T03:35:30.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068538899999999996 2023-10-20T03:35:30.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.405783+0000", 2023-10-20T03:35:30.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5658000000000001e-05 2023-10-20T03:35:30.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.469 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.470 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 44/31 ECSubWrite(tid=97, reqid=client.4570.0:341, at_version=44'19, trim_to=0'0, roll_forward_to=43'17))", 2023-10-20T03:35:30.470 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.402172+0000", 2023-10-20T03:35:30.470 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8934248419999999, 2023-10-20T03:35:30.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0051350049999999998, 2023-10-20T03:35:30.471 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.471 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.471 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.402172+0000", 2023-10-20T03:35:30.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.402172+0000", 2023-10-20T03:35:30.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.402174+0000", 2023-10-20T03:35:30.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9539999999999998e-06 2023-10-20T03:35:30.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.402229+0000", 2023-10-20T03:35:30.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4981000000000001e-05 2023-10-20T03:35:30.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.402260+0000", 2023-10-20T03:35:30.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0433000000000002e-05 2023-10-20T03:35:30.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.402281+0000", 2023-10-20T03:35:30.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1668999999999999e-05 2023-10-20T03:35:30.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.402461+0000", 2023-10-20T03:35:30.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000180142 2023-10-20T03:35:30.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.402488+0000", 2023-10-20T03:35:30.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6381999999999999e-05 2023-10-20T03:35:30.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.407236+0000", 2023-10-20T03:35:30.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047483660000000004 2023-10-20T03:35:30.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.407307+0000", 2023-10-20T03:35:30.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1077999999999999e-05 2023-10-20T03:35:30.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.481 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.482 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/31 ECSubWrite(tid=21, reqid=client.4570.0:326, at_version=44'12, trim_to=0'0, roll_forward_to=44'9))", 2023-10-20T03:35:30.482 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.403439+0000", 2023-10-20T03:35:30.482 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8921582369999999, 2023-10-20T03:35:30.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038625270000000001, 2023-10-20T03:35:30.483 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.483 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.483 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.403439+0000", 2023-10-20T03:35:30.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.403439+0000", 2023-10-20T03:35:30.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.403441+0000", 2023-10-20T03:35:30.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7600000000000001e-06 2023-10-20T03:35:30.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.403494+0000", 2023-10-20T03:35:30.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3108e-05 2023-10-20T03:35:30.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.403527+0000", 2023-10-20T03:35:30.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3169e-05 2023-10-20T03:35:30.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.403550+0000", 2023-10-20T03:35:30.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3442999999999999e-05 2023-10-20T03:35:30.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.403667+0000", 2023-10-20T03:35:30.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011712700000000001 2023-10-20T03:35:30.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.403681+0000", 2023-10-20T03:35:30.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.401e-05 2023-10-20T03:35:30.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.407236+0000", 2023-10-20T03:35:30.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035542550000000001 2023-10-20T03:35:30.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.407301+0000", 2023-10-20T03:35:30.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5654999999999994e-05 2023-10-20T03:35:30.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.493 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.494 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 44/31 ECSubWrite(tid=98, reqid=client.4570.0:341, at_version=0'0, trim_to=0'0, roll_forward_to=44'19))", 2023-10-20T03:35:30.494 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.409225+0000", 2023-10-20T03:35:30.494 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8863719269999999, 2023-10-20T03:35:30.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003815517, 2023-10-20T03:35:30.495 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.495 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.495 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409225+0000", 2023-10-20T03:35:30.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409225+0000", 2023-10-20T03:35:30.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409227+0000", 2023-10-20T03:35:30.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.443e-06 2023-10-20T03:35:30.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409252+0000", 2023-10-20T03:35:30.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5279999999999999e-05 2023-10-20T03:35:30.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409280+0000", 2023-10-20T03:35:30.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7902000000000002e-05 2023-10-20T03:35:30.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409302+0000", 2023-10-20T03:35:30.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2555999999999999e-05 2023-10-20T03:35:30.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409493+0000", 2023-10-20T03:35:30.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019070799999999999 2023-10-20T03:35:30.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409511+0000", 2023-10-20T03:35:30.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8369999999999999e-05 2023-10-20T03:35:30.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.412961+0000", 2023-10-20T03:35:30.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034499470000000001 2023-10-20T03:35:30.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.413041+0000", 2023-10-20T03:35:30.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9311e-05 2023-10-20T03:35:30.505 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.505 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.505 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.506 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s3 44/31 ECSubWrite(tid=99, reqid=client.4570.0:342, at_version=44'34, trim_to=0'0, roll_forward_to=44'33))", 2023-10-20T03:35:30.506 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.411593+0000", 2023-10-20T03:35:30.506 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.884003619, 2023-10-20T03:35:30.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039951250000000004, 2023-10-20T03:35:30.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.507 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.411593+0000", 2023-10-20T03:35:30.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.411593+0000", 2023-10-20T03:35:30.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.411595+0000", 2023-10-20T03:35:30.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.536e-06 2023-10-20T03:35:30.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.411717+0000", 2023-10-20T03:35:30.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012207 2023-10-20T03:35:30.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.411751+0000", 2023-10-20T03:35:30.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3788999999999998e-05 2023-10-20T03:35:30.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.411775+0000", 2023-10-20T03:35:30.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4275999999999999e-05 2023-10-20T03:35:30.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.411936+0000", 2023-10-20T03:35:30.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016141500000000001 2023-10-20T03:35:30.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.411961+0000", 2023-10-20T03:35:30.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4627e-05 2023-10-20T03:35:30.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.415524+0000", 2023-10-20T03:35:30.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035633399999999999 2023-10-20T03:35:30.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.415589+0000", 2023-10-20T03:35:30.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4072000000000006e-05 2023-10-20T03:35:30.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.517 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.517 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.518 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/31 ECSubWrite(tid=22, reqid=client.4570.0:326, at_version=0'0, trim_to=0'0, roll_forward_to=44'12))", 2023-10-20T03:35:30.518 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.413356+0000", 2023-10-20T03:35:30.518 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8822405849999999, 2023-10-20T03:35:30.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0062475769999999998, 2023-10-20T03:35:30.518 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.519 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.519 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.413356+0000", 2023-10-20T03:35:30.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.413356+0000", 2023-10-20T03:35:30.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.413358+0000", 2023-10-20T03:35:30.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8369999999999999e-06 2023-10-20T03:35:30.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.413393+0000", 2023-10-20T03:35:30.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4579000000000002e-05 2023-10-20T03:35:30.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.413422+0000", 2023-10-20T03:35:30.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9028999999999999e-05 2023-10-20T03:35:30.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.413465+0000", 2023-10-20T03:35:30.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2818999999999998e-05 2023-10-20T03:35:30.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.413620+0000", 2023-10-20T03:35:30.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000155532 2023-10-20T03:35:30.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.413634+0000", 2023-10-20T03:35:30.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3686000000000001e-05 2023-10-20T03:35:30.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.419507+0000", 2023-10-20T03:35:30.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058726070000000002 2023-10-20T03:35:30.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.419604+0000", 2023-10-20T03:35:30.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7487999999999999e-05 2023-10-20T03:35:30.528 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.528 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.529 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s3 44/31 ECSubWrite(tid=100, reqid=client.4570.0:342, at_version=0'0, trim_to=0'0, roll_forward_to=44'34))", 2023-10-20T03:35:30.529 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.416627+0000", 2023-10-20T03:35:30.529 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8789702459999997, 2023-10-20T03:35:30.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038428809999999998, 2023-10-20T03:35:30.530 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:30.530 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:30.530 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:30.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:30.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.416627+0000", 2023-10-20T03:35:30.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:30.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.416627+0000", 2023-10-20T03:35:30.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:30.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:30.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.416628+0000", 2023-10-20T03:35:30.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3599999999999999e-06 2023-10-20T03:35:30.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:30.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.416653+0000", 2023-10-20T03:35:30.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5179000000000001e-05 2023-10-20T03:35:30.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:30.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.416681+0000", 2023-10-20T03:35:30.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7251999999999999e-05 2023-10-20T03:35:30.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:30.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.416695+0000", 2023-10-20T03:35:30.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4803999999999999e-05 2023-10-20T03:35:30.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:30.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.416813+0000", 2023-10-20T03:35:30.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117593 2023-10-20T03:35:30.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:30.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.416836+0000", 2023-10-20T03:35:30.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2629e-05 2023-10-20T03:35:30.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:30.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.420393+0000", 2023-10-20T03:35:30.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003557499 2023-10-20T03:35:30.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:30.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:30.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:30.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.420470+0000", 2023-10-20T03:35:30.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6564999999999996e-05 2023-10-20T03:35:30.540 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.541 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.541 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.541 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:30.541 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:30.541 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:30.542 INFO:tasks.rados.rados.0.smithi170.stdout:153: done (5 left) 2023-10-20T03:35:30.543 INFO:tasks.rados.rados.0.smithi170.stdout:154: done (4 left) 2023-10-20T03:35:30.543 INFO:tasks.rados.rados.0.smithi170.stdout:155: done (3 left) 2023-10-20T03:35:30.543 INFO:tasks.rados.rados.0.smithi170.stdout:156: done (2 left) 2023-10-20T03:35:30.543 INFO:tasks.rados.rados.0.smithi170.stdout:157: done (1 left) 2023-10-20T03:35:30.544 INFO:tasks.rados.rados.0.smithi170.stdout:158: done (0 left) 2023-10-20T03:35:30.544 INFO:tasks.rados.rados.0.smithi170.stdout:159: copy_from oid 4 from oid 34 current snap is 11 2023-10-20T03:35:30.544 INFO:tasks.rados.rados.0.smithi170.stdout:160: snap_remove snap 6 2023-10-20T03:35:30.545 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-10-20T03:35:30.546 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:30.541+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:30.644 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:30.641+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:30.678 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2023-10-20T03:35:30.679 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:30.694 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-10-20T03:35:30.745 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:30.740+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:30.820 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:30.820 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2023-10-20T03:35:30.821 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2023-10-20T03:35:30.821 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:30.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-10-20T03:35:30.845 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:30.846+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:30.947 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:30.945+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:31.004 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:31.004 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2023-10-20T03:35:31.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2023-10-20T03:35:31.004 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2023-10-20T03:35:31.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.005 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.2s5 36/31 ECSubWrite(tid=21, reqid=client.4570.0:198, at_version=36'13, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:31.005 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:18.265428+0000", 2023-10-20T03:35:31.005 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.736193706, 2023-10-20T03:35:31.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018643989, 2023-10-20T03:35:31.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.006 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.006 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.265428+0000", 2023-10-20T03:35:31.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.265428+0000", 2023-10-20T03:35:31.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.265429+0000", 2023-10-20T03:35:31.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6000000000000004e-07 2023-10-20T03:35:31.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.265776+0000", 2023-10-20T03:35:31.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000347752 2023-10-20T03:35:31.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.265839+0000", 2023-10-20T03:35:31.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2898000000000007e-05 2023-10-20T03:35:31.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.265869+0000", 2023-10-20T03:35:31.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9575999999999999e-05 2023-10-20T03:35:31.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.266078+0000", 2023-10-20T03:35:31.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020933599999999999 2023-10-20T03:35:31.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.266106+0000", 2023-10-20T03:35:31.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7818999999999999e-05 2023-10-20T03:35:31.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.283977+0000", 2023-10-20T03:35:31.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017870994000000001 2023-10-20T03:35:31.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.284072+0000", 2023-10-20T03:35:31.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5054000000000002e-05 2023-10-20T03:35:31.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.017 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.2s5 36/31 ECSubWrite(tid=22, reqid=client.4570.0:199, at_version=36'14, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:31.017 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:18.284872+0000", 2023-10-20T03:35:31.017 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.716750338000001, 2023-10-20T03:35:31.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010083263, 2023-10-20T03:35:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.284872+0000", 2023-10-20T03:35:31.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.284872+0000", 2023-10-20T03:35:31.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.284874+0000", 2023-10-20T03:35:31.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0770000000000002e-06 2023-10-20T03:35:31.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.285100+0000", 2023-10-20T03:35:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022611199999999999 2023-10-20T03:35:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.285140+0000", 2023-10-20T03:35:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0408999999999999e-05 2023-10-20T03:35:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.285166+0000", 2023-10-20T03:35:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5644999999999999e-05 2023-10-20T03:35:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.285319+0000", 2023-10-20T03:35:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015350499999999999 2023-10-20T03:35:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.285339+0000", 2023-10-20T03:35:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9959e-05 2023-10-20T03:35:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.294858+0000", 2023-10-20T03:35:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.009519062 2023-10-20T03:35:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.294955+0000", 2023-10-20T03:35:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6494000000000001e-05 2023-10-20T03:35:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.2s5 36/31 ECSubWrite(tid=23, reqid=client.4570.0:200, at_version=36'15, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:18.293533+0000", 2023-10-20T03:35:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.708088504999999, 2023-10-20T03:35:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021368200000000002, 2023-10-20T03:35:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.293533+0000", 2023-10-20T03:35:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.293533+0000", 2023-10-20T03:35:31.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.293535+0000", 2023-10-20T03:35:31.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8729999999999999e-06 2023-10-20T03:35:31.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.293599+0000", 2023-10-20T03:35:31.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3893999999999995e-05 2023-10-20T03:35:31.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.293643+0000", 2023-10-20T03:35:31.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3714999999999997e-05 2023-10-20T03:35:31.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.293668+0000", 2023-10-20T03:35:31.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5086e-05 2023-10-20T03:35:31.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.293815+0000", 2023-10-20T03:35:31.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014668600000000001 2023-10-20T03:35:31.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.293838+0000", 2023-10-20T03:35:31.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3853999999999999e-05 2023-10-20T03:35:31.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.295601+0000", 2023-10-20T03:35:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017625939999999999 2023-10-20T03:35:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.295670+0000", 2023-10-20T03:35:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9117999999999998e-05 2023-10-20T03:35:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.2s5 36/31 ECSubWrite(tid=24, reqid=client.4570.0:200, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-10-20T03:35:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:18.298000+0000", 2023-10-20T03:35:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.703621377999999, 2023-10-20T03:35:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018117039999999999, 2023-10-20T03:35:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.298000+0000", 2023-10-20T03:35:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.298000+0000", 2023-10-20T03:35:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.298002+0000", 2023-10-20T03:35:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8330000000000001e-06 2023-10-20T03:35:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.298026+0000", 2023-10-20T03:35:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3453000000000001e-05 2023-10-20T03:35:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.298057+0000", 2023-10-20T03:35:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0939000000000003e-05 2023-10-20T03:35:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.298072+0000", 2023-10-20T03:35:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5455999999999999e-05 2023-10-20T03:35:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.298201+0000", 2023-10-20T03:35:31.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012880199999999999 2023-10-20T03:35:31.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.298216+0000", 2023-10-20T03:35:31.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5273000000000001e-05 2023-10-20T03:35:31.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.299705+0000", 2023-10-20T03:35:31.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001488861 2023-10-20T03:35:31.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:18.299812+0000", 2023-10-20T03:35:31.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107087 2023-10-20T03:35:31.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s5 38/31 ECSubWrite(tid=63, reqid=client.4570.0:216, at_version=38'7, trim_to=0'0, roll_forward_to=34'5))", 2023-10-20T03:35:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:20.272886+0000", 2023-10-20T03:35:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.72873558, 2023-10-20T03:35:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040593419999999996, 2023-10-20T03:35:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.272886+0000", 2023-10-20T03:35:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.272886+0000", 2023-10-20T03:35:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.272887+0000", 2023-10-20T03:35:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3000000000000001e-07 2023-10-20T03:35:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.272954+0000", 2023-10-20T03:35:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7744999999999996e-05 2023-10-20T03:35:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.272978+0000", 2023-10-20T03:35:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3972e-05 2023-10-20T03:35:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.273002+0000", 2023-10-20T03:35:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3226000000000001e-05 2023-10-20T03:35:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.273228+0000", 2023-10-20T03:35:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022621699999999999 2023-10-20T03:35:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.273256+0000", 2023-10-20T03:35:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7849000000000001e-05 2023-10-20T03:35:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.276845+0000", 2023-10-20T03:35:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003588833 2023-10-20T03:35:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.276946+0000", 2023-10-20T03:35:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010107000000000001 2023-10-20T03:35:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.7s5 38/31 ECSubWrite(tid=64, reqid=client.4570.0:216, at_version=0'0, trim_to=0'0, roll_forward_to=38'7))", 2023-10-20T03:35:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:20.278530+0000", 2023-10-20T03:35:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.723091946, 2023-10-20T03:35:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030310559999999999, 2023-10-20T03:35:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.278530+0000", 2023-10-20T03:35:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.278530+0000", 2023-10-20T03:35:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.278532+0000", 2023-10-20T03:35:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9759999999999998e-06 2023-10-20T03:35:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.278568+0000", 2023-10-20T03:35:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6136000000000001e-05 2023-10-20T03:35:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.278599+0000", 2023-10-20T03:35:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0799000000000001e-05 2023-10-20T03:35:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.278626+0000", 2023-10-20T03:35:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6699e-05 2023-10-20T03:35:31.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.278773+0000", 2023-10-20T03:35:31.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147565 2023-10-20T03:35:31.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.278797+0000", 2023-10-20T03:35:31.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3873e-05 2023-10-20T03:35:31.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.281489+0000", 2023-10-20T03:35:31.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026917270000000001 2023-10-20T03:35:31.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:20.281561+0000", 2023-10-20T03:35:31.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2280999999999995e-05 2023-10-20T03:35:31.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.075 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.075 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:31.076 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.330272+0000", 2023-10-20T03:35:31.076 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.671349728, 2023-10-20T03:35:31.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035047879999999997, 2023-10-20T03:35:31.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.076 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.076 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330272+0000", 2023-10-20T03:35:31.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330272+0000", 2023-10-20T03:35:31.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330273+0000", 2023-10-20T03:35:31.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6300000000000005e-07 2023-10-20T03:35:31.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330709+0000", 2023-10-20T03:35:31.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043642900000000001 2023-10-20T03:35:31.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330745+0000", 2023-10-20T03:35:31.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6161999999999997e-05 2023-10-20T03:35:31.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330770+0000", 2023-10-20T03:35:31.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5151999999999999e-05 2023-10-20T03:35:31.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.331008+0000", 2023-10-20T03:35:31.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023709800000000001 2023-10-20T03:35:31.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.331035+0000", 2023-10-20T03:35:31.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7178999999999998e-05 2023-10-20T03:35:31.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365221+0000", 2023-10-20T03:35:31.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034186753 2023-10-20T03:35:31.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365320+0000", 2023-10-20T03:35:31.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8543999999999997e-05 2023-10-20T03:35:31.086 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.086 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.086 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.087 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 43/31 ECSubWrite(tid=81, reqid=client.4570.0:307, at_version=43'15, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:31.087 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.343425+0000", 2023-10-20T03:35:31.087 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.6581971800000002, 2023-10-20T03:35:31.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021973126999999999, 2023-10-20T03:35:31.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.088 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.088 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343425+0000", 2023-10-20T03:35:31.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343425+0000", 2023-10-20T03:35:31.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343427+0000", 2023-10-20T03:35:31.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3999999999999999e-06 2023-10-20T03:35:31.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343627+0000", 2023-10-20T03:35:31.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000200099 2023-10-20T03:35:31.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343664+0000", 2023-10-20T03:35:31.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7165000000000001e-05 2023-10-20T03:35:31.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343690+0000", 2023-10-20T03:35:31.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5732999999999999e-05 2023-10-20T03:35:31.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.345198+0000", 2023-10-20T03:35:31.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001507743 2023-10-20T03:35:31.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.345227+0000", 2023-10-20T03:35:31.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9546000000000001e-05 2023-10-20T03:35:31.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365338+0000", 2023-10-20T03:35:31.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020110209 2023-10-20T03:35:31.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365398+0000", 2023-10-20T03:35:31.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0232000000000002e-05 2023-10-20T03:35:31.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.097 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.098 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 43/31 ECSubWrite(tid=83, reqid=client.4570.0:308, at_version=43'16, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:31.098 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.356060+0000", 2023-10-20T03:35:31.098 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.6455616470000001, 2023-10-20T03:35:31.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010314891, 2023-10-20T03:35:31.099 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.099 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.099 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356060+0000", 2023-10-20T03:35:31.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356060+0000", 2023-10-20T03:35:31.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356062+0000", 2023-10-20T03:35:31.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7600000000000001e-06 2023-10-20T03:35:31.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356118+0000", 2023-10-20T03:35:31.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6452000000000001e-05 2023-10-20T03:35:31.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356149+0000", 2023-10-20T03:35:31.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0533000000000001e-05 2023-10-20T03:35:31.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356171+0000", 2023-10-20T03:35:31.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2011999999999999e-05 2023-10-20T03:35:31.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.362986+0000", 2023-10-20T03:35:31.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0068145380000000002 2023-10-20T03:35:31.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.363013+0000", 2023-10-20T03:35:31.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7919000000000002e-05 2023-10-20T03:35:31.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.366314+0000", 2023-10-20T03:35:31.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033007079999999999 2023-10-20T03:35:31.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.366375+0000", 2023-10-20T03:35:31.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0968999999999999e-05 2023-10-20T03:35:31.108 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.108 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.109 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.109 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 43/31 ECSubWrite(tid=84, reqid=client.4570.0:309, at_version=43'17, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:31.109 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.359636+0000", 2023-10-20T03:35:31.109 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.6419854689999998, 2023-10-20T03:35:31.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074022879999999999, 2023-10-20T03:35:31.110 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.110 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.110 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359636+0000", 2023-10-20T03:35:31.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359636+0000", 2023-10-20T03:35:31.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359639+0000", 2023-10-20T03:35:31.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2359999999999999e-06 2023-10-20T03:35:31.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359682+0000", 2023-10-20T03:35:31.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3785999999999997e-05 2023-10-20T03:35:31.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359724+0000", 2023-10-20T03:35:31.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1572000000000001e-05 2023-10-20T03:35:31.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359746+0000", 2023-10-20T03:35:31.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1543e-05 2023-10-20T03:35:31.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365511+0000", 2023-10-20T03:35:31.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057650959999999999 2023-10-20T03:35:31.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365542+0000", 2023-10-20T03:35:31.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1355e-05 2023-10-20T03:35:31.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.366997+0000", 2023-10-20T03:35:31.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014553330000000001 2023-10-20T03:35:31.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.367039+0000", 2023-10-20T03:35:31.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1366999999999999e-05 2023-10-20T03:35:31.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.120 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.120 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 43/31 ECSubWrite(tid=86, reqid=client.4570.0:309, at_version=0'0, trim_to=0'0, roll_forward_to=43'17))", 2023-10-20T03:35:31.120 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.377314+0000", 2023-10-20T03:35:31.121 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.6243081579999998, 2023-10-20T03:35:31.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020652119999999999, 2023-10-20T03:35:31.121 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.121 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.121 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.377314+0000", 2023-10-20T03:35:31.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.377314+0000", 2023-10-20T03:35:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.377316+0000", 2023-10-20T03:35:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1399999999999998e-06 2023-10-20T03:35:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.377348+0000", 2023-10-20T03:35:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1769e-05 2023-10-20T03:35:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.377389+0000", 2023-10-20T03:35:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1295999999999999e-05 2023-10-20T03:35:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.377412+0000", 2023-10-20T03:35:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3479e-05 2023-10-20T03:35:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.377572+0000", 2023-10-20T03:35:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015986200000000001 2023-10-20T03:35:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.377597+0000", 2023-10-20T03:35:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4902999999999999e-05 2023-10-20T03:35:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.379283+0000", 2023-10-20T03:35:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016857459999999999 2023-10-20T03:35:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.379379+0000", 2023-10-20T03:35:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6016999999999999e-05 2023-10-20T03:35:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.1s3 43/31 ECSubRead(tid=87, to_read={3:8d0c2319:::smithi17039300-31:head=0,4096,0}, subchunks={3:8d0c2319:::smithi17039300-31:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.378753+0000", 2023-10-20T03:35:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.6228693500000002, 2023-10-20T03:35:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043638699999999999, 2023-10-20T03:35:31.132 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.378753+0000", 2023-10-20T03:35:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.378753+0000", 2023-10-20T03:35:31.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.378754+0000", 2023-10-20T03:35:31.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.556e-06 2023-10-20T03:35:31.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.378772+0000", 2023-10-20T03:35:31.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8253999999999998e-05 2023-10-20T03:35:31.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.378811+0000", 2023-10-20T03:35:31.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8588999999999999e-05 2023-10-20T03:35:31.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.378836+0000", 2023-10-20T03:35:31.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5508999999999999e-05 2023-10-20T03:35:31.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.378984+0000", 2023-10-20T03:35:31.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147632 2023-10-20T03:35:31.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.379189+0000", 2023-10-20T03:35:31.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000204847 2023-10-20T03:35:31.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.140 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.141 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "pg_update_log_missing(3.9s2 epoch 44/31 rep_tid 35 entries 44'22 (0'0) error 3:97c0e862:::smithi17039300-18:head by client.4570.0:321 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 39'20)", 2023-10-20T03:35:31.141 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.335125+0000", 2023-10-20T03:35:31.142 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6664972049999998, 2023-10-20T03:35:31.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022370509999999999, 2023-10-20T03:35:31.142 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.142 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.142 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.335125+0000", 2023-10-20T03:35:31.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.335125+0000", 2023-10-20T03:35:31.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.335125+0000", 2023-10-20T03:35:31.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8299999999999997e-07 2023-10-20T03:35:31.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.335140+0000", 2023-10-20T03:35:31.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4473e-05 2023-10-20T03:35:31.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.335191+0000", 2023-10-20T03:35:31.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1822e-05 2023-10-20T03:35:31.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.335230+0000", 2023-10-20T03:35:31.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8254999999999999e-05 2023-10-20T03:35:31.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.335588+0000", 2023-10-20T03:35:31.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000358084 2023-10-20T03:35:31.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.337362+0000", 2023-10-20T03:35:31.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001773934 2023-10-20T03:35:31.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.150 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.151 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.2s5 44/31 ECSubWrite(tid=29, reqid=client.4570.0:320, at_version=44'17, trim_to=0'0, roll_forward_to=36'15))", 2023-10-20T03:35:31.151 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.342446+0000", 2023-10-20T03:35:31.152 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6591757920000001, 2023-10-20T03:35:31.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041511780000000002, 2023-10-20T03:35:31.152 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.152 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.342446+0000", 2023-10-20T03:35:31.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.342446+0000", 2023-10-20T03:35:31.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.342448+0000", 2023-10-20T03:35:31.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4470000000000001e-06 2023-10-20T03:35:31.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.342693+0000", 2023-10-20T03:35:31.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024573299999999999 2023-10-20T03:35:31.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.342728+0000", 2023-10-20T03:35:31.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5206000000000001e-05 2023-10-20T03:35:31.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.342817+0000", 2023-10-20T03:35:31.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8710999999999998e-05 2023-10-20T03:35:31.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.343136+0000", 2023-10-20T03:35:31.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00031868100000000002 2023-10-20T03:35:31.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.343190+0000", 2023-10-20T03:35:31.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3865e-05 2023-10-20T03:35:31.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.346502+0000", 2023-10-20T03:35:31.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003312658 2023-10-20T03:35:31.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.346597+0000", 2023-10-20T03:35:31.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4877000000000007e-05 2023-10-20T03:35:31.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.162 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.162 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.2s5 44/31 ECSubWrite(tid=30, reqid=client.4570.0:320, at_version=0'0, trim_to=0'0, roll_forward_to=44'17))", 2023-10-20T03:35:31.163 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.350174+0000", 2023-10-20T03:35:31.163 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6514474589999999, 2023-10-20T03:35:31.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001760548, 2023-10-20T03:35:31.163 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.163 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.164 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.350174+0000", 2023-10-20T03:35:31.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.350174+0000", 2023-10-20T03:35:31.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.350176+0000", 2023-10-20T03:35:31.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0729999999999999e-06 2023-10-20T03:35:31.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.350205+0000", 2023-10-20T03:35:31.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8886000000000001e-05 2023-10-20T03:35:31.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.350248+0000", 2023-10-20T03:35:31.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2635000000000001e-05 2023-10-20T03:35:31.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.350272+0000", 2023-10-20T03:35:31.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3533e-05 2023-10-20T03:35:31.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.350467+0000", 2023-10-20T03:35:31.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019521499999999999 2023-10-20T03:35:31.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.350484+0000", 2023-10-20T03:35:31.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7533e-05 2023-10-20T03:35:31.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.351836+0000", 2023-10-20T03:35:31.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001351969 2023-10-20T03:35:31.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.351935+0000", 2023-10-20T03:35:31.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8703999999999996e-05 2023-10-20T03:35:31.173 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.173 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.173 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.174 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 44/31 ECSubWrite(tid=97, reqid=client.4570.0:341, at_version=44'19, trim_to=0'0, roll_forward_to=43'17))", 2023-10-20T03:35:31.174 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.401595+0000", 2023-10-20T03:35:31.174 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6000264839999998, 2023-10-20T03:35:31.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031569520000000002, 2023-10-20T03:35:31.175 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.175 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.175 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.401595+0000", 2023-10-20T03:35:31.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.401595+0000", 2023-10-20T03:35:31.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.401596+0000", 2023-10-20T03:35:31.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3000000000000001e-07 2023-10-20T03:35:31.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.401659+0000", 2023-10-20T03:35:31.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3467999999999996e-05 2023-10-20T03:35:31.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.401696+0000", 2023-10-20T03:35:31.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6511999999999999e-05 2023-10-20T03:35:31.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.401721+0000", 2023-10-20T03:35:31.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4749e-05 2023-10-20T03:35:31.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.401847+0000", 2023-10-20T03:35:31.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012667399999999999 2023-10-20T03:35:31.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.401866+0000", 2023-10-20T03:35:31.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8966000000000001e-05 2023-10-20T03:35:31.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.404684+0000", 2023-10-20T03:35:31.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028178679999999998 2023-10-20T03:35:31.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.404752+0000", 2023-10-20T03:35:31.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8184999999999999e-05 2023-10-20T03:35:31.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.184 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.185 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 44/31 ECSubWrite(tid=98, reqid=client.4570.0:341, at_version=0'0, trim_to=0'0, roll_forward_to=44'19))", 2023-10-20T03:35:31.185 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.409768+0000", 2023-10-20T03:35:31.185 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.5918538069999997, 2023-10-20T03:35:31.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024689479999999999, 2023-10-20T03:35:31.185 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.186 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.186 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409768+0000", 2023-10-20T03:35:31.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409768+0000", 2023-10-20T03:35:31.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409770+0000", 2023-10-20T03:35:31.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.956e-06 2023-10-20T03:35:31.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409799+0000", 2023-10-20T03:35:31.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8822999999999998e-05 2023-10-20T03:35:31.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409842+0000", 2023-10-20T03:35:31.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3417999999999997e-05 2023-10-20T03:35:31.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.409866+0000", 2023-10-20T03:35:31.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.421e-05 2023-10-20T03:35:31.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410055+0000", 2023-10-20T03:35:31.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018865099999999999 2023-10-20T03:35:31.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410080+0000", 2023-10-20T03:35:31.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5355999999999999e-05 2023-10-20T03:35:31.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.412159+0000", 2023-10-20T03:35:31.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020785220000000002 2023-10-20T03:35:31.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.412237+0000", 2023-10-20T03:35:31.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8011999999999998e-05 2023-10-20T03:35:31.195 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.195 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.195 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.196 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.9s2 44/31 ECSubRead(tid=36, to_read={3:93433de7:::smithi17039300-36:head=0,454656,0}, subchunks={3:93433de7:::smithi17039300-36:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:31.196 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.410061+0000", 2023-10-20T03:35:31.196 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.5915608959999998, 2023-10-20T03:35:31.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020886519999999999, 2023-10-20T03:35:31.197 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.197 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.197 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410061+0000", 2023-10-20T03:35:31.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410061+0000", 2023-10-20T03:35:31.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410063+0000", 2023-10-20T03:35:31.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2199999999999999e-06 2023-10-20T03:35:31.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410084+0000", 2023-10-20T03:35:31.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1290000000000001e-05 2023-10-20T03:35:31.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410121+0000", 2023-10-20T03:35:31.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6628999999999998e-05 2023-10-20T03:35:31.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410147+0000", 2023-10-20T03:35:31.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5689000000000001e-05 2023-10-20T03:35:31.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410322+0000", 2023-10-20T03:35:31.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017566500000000001 2023-10-20T03:35:31.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.412150+0000", 2023-10-20T03:35:31.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018271590000000001 2023-10-20T03:35:31.205 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.205 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.205 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.205 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.9s2 44/31 ECSubRead(tid=37, to_read={3:93433de7:::smithi17039300-36:head=0,454656,0}, subchunks={3:93433de7:::smithi17039300-36:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:31.206 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.410736+0000", 2023-10-20T03:35:31.206 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.5908862140000002, 2023-10-20T03:35:31.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019677779999999999, 2023-10-20T03:35:31.206 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.206 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.207 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410736+0000", 2023-10-20T03:35:31.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410736+0000", 2023-10-20T03:35:31.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410737+0000", 2023-10-20T03:35:31.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.66e-06 2023-10-20T03:35:31.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410757+0000", 2023-10-20T03:35:31.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0152e-05 2023-10-20T03:35:31.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410791+0000", 2023-10-20T03:35:31.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3065999999999998e-05 2023-10-20T03:35:31.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.410817+0000", 2023-10-20T03:35:31.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6526e-05 2023-10-20T03:35:31.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.412249+0000", 2023-10-20T03:35:31.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00143238 2023-10-20T03:35:31.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.412703+0000", 2023-10-20T03:35:31.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00045399399999999999 2023-10-20T03:35:31.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.214 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.215 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpRead(3.9s2 44/31 ECSubRead(tid=38, to_read={3:93433de7:::smithi17039300-36:head=0,454656,0}, subchunks={3:93433de7:::smithi17039300-36:head=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:31.215 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.412846+0000", 2023-10-20T03:35:31.216 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.5887759399999997, 2023-10-20T03:35:31.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093467400000000003, 2023-10-20T03:35:31.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.216 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.216 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.412846+0000", 2023-10-20T03:35:31.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.412846+0000", 2023-10-20T03:35:31.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.412847+0000", 2023-10-20T03:35:31.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5069999999999999e-06 2023-10-20T03:35:31.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.412871+0000", 2023-10-20T03:35:31.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4036000000000002e-05 2023-10-20T03:35:31.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.413060+0000", 2023-10-20T03:35:31.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018824799999999999 2023-10-20T03:35:31.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.413089+0000", 2023-10-20T03:35:31.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9266000000000001e-05 2023-10-20T03:35:31.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.413361+0000", 2023-10-20T03:35:31.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027199899999999998 2023-10-20T03:35:31.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.413781+0000", 2023-10-20T03:35:31.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041961800000000002 2023-10-20T03:35:31.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.224 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.225 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.225 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:31.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-10-20T03:35:31.227 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:31.220+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:31.327 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:31.320+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:31.380 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-10-20T03:35:31.382 INFO:tasks.rados.rados.0.smithi170.stdout:160: done (1 left) 2023-10-20T03:35:31.382 INFO:tasks.rados.rados.0.smithi170.stdout:161: read oid 39 snap 9 2023-10-20T03:35:31.382 INFO:tasks.rados.rados.0.smithi170.stdout:161: expect (ObjNum 58 snap 0 seq_num 58) 2023-10-20T03:35:31.382 INFO:tasks.rados.rados.0.smithi170.stdout:162: copy_from oid 49 from oid 43 current snap is 11 2023-10-20T03:35:31.382 INFO:tasks.rados.rados.0.smithi170.stdout:159: finishing copy_from racing read to smithi17039300-4 2023-10-20T03:35:31.383 INFO:tasks.rados.rados.0.smithi170.stdout:159: finishing copy_from to smithi17039300-4 2023-10-20T03:35:31.383 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 21 (ObjNum 81 snap 8 seq_num 81) dirty exists 2023-10-20T03:35:31.383 INFO:tasks.rados.rados.0.smithi170.stdout:159: done (2 left) 2023-10-20T03:35:31.383 INFO:tasks.rados.rados.0.smithi170.stdout:163: delete oid 29 current snap is 11 2023-10-20T03:35:31.402 INFO:tasks.rados.rados.0.smithi170.stdout:163: done (2 left) 2023-10-20T03:35:31.402 INFO:tasks.rados.rados.0.smithi170.stdout:164: snap_remove snap 3 2023-10-20T03:35:31.418 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:31.418 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2023-10-20T03:35:31.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.419 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 49/31 ECSubWrite(tid=121, reqid=osd.13.0:119, at_version=0'0, trim_to=0'0, roll_forward_to=49'24))", 2023-10-20T03:35:31.419 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.417090+0000", 2023-10-20T03:35:31.419 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.001784924, 2023-10-20T03:35:31.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001808175, 2023-10-20T03:35:31.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.420 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.417090+0000", 2023-10-20T03:35:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.417090+0000", 2023-10-20T03:35:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.417091+0000", 2023-10-20T03:35:31.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0299999999999998e-07 2023-10-20T03:35:31.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.417103+0000", 2023-10-20T03:35:31.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2595999999999999e-05 2023-10-20T03:35:31.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.417120+0000", 2023-10-20T03:35:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6554000000000001e-05 2023-10-20T03:35:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.417129+0000", 2023-10-20T03:35:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5349999999999993e-06 2023-10-20T03:35:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.417198+0000", 2023-10-20T03:35:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8709999999999998e-05 2023-10-20T03:35:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.417208+0000", 2023-10-20T03:35:31.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8109999999999997e-06 2023-10-20T03:35:31.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.428 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.429 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2023-10-20T03:35:31.429 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2023-10-20T03:35:31.429 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:31.430 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:31.429+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:31.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-10-20T03:35:31.532 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:31.530+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:31.602 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:31.603 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:31.603 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2023-10-20T03:35:31.603 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2023-10-20T03:35:31.603 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:31.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-10-20T03:35:31.664 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:31.660+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:31.733 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:31.730+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:31.746 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:31.747 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2023-10-20T03:35:31.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2023-10-20T03:35:31.747 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2023-10-20T03:35:31.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.747 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es1 37/31 ECSubWrite(tid=23, reqid=client.4570.0:208, at_version=37'14, trim_to=0'0, roll_forward_to=36'12))", 2023-10-20T03:35:31.748 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:19.273182+0000", 2023-10-20T03:35:31.748 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.470880205, 2023-10-20T03:35:31.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016185486999999998, 2023-10-20T03:35:31.748 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.748 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.749 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.273182+0000", 2023-10-20T03:35:31.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.273182+0000", 2023-10-20T03:35:31.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.273182+0000", 2023-10-20T03:35:31.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5999999999999999e-07 2023-10-20T03:35:31.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.273913+0000", 2023-10-20T03:35:31.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073112000000000003 2023-10-20T03:35:31.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.273946+0000", 2023-10-20T03:35:31.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2792e-05 2023-10-20T03:35:31.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.273967+0000", 2023-10-20T03:35:31.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1279000000000001e-05 2023-10-20T03:35:31.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.274135+0000", 2023-10-20T03:35:31.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167412 2023-10-20T03:35:31.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.274154+0000", 2023-10-20T03:35:31.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8872000000000002e-05 2023-10-20T03:35:31.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.289287+0000", 2023-10-20T03:35:31.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015133604 2023-10-20T03:35:31.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.289367+0000", 2023-10-20T03:35:31.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0048000000000003e-05 2023-10-20T03:35:31.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.758 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.759 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.759 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es1 37/31 ECSubWrite(tid=24, reqid=client.4570.0:209, at_version=37'15, trim_to=0'0, roll_forward_to=36'12))", 2023-10-20T03:35:31.759 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:19.289716+0000", 2023-10-20T03:35:31.760 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.454345846000001, 2023-10-20T03:35:31.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080323180000000001, 2023-10-20T03:35:31.760 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.760 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.760 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.289716+0000", 2023-10-20T03:35:31.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.289716+0000", 2023-10-20T03:35:31.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.289718+0000", 2023-10-20T03:35:31.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7829999999999999e-06 2023-10-20T03:35:31.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.289900+0000", 2023-10-20T03:35:31.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000182344 2023-10-20T03:35:31.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.289928+0000", 2023-10-20T03:35:31.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7453e-05 2023-10-20T03:35:31.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.289942+0000", 2023-10-20T03:35:31.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3882e-05 2023-10-20T03:35:31.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.290085+0000", 2023-10-20T03:35:31.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143036 2023-10-20T03:35:31.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.290102+0000", 2023-10-20T03:35:31.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7102e-05 2023-10-20T03:35:31.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.297691+0000", 2023-10-20T03:35:31.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0075889959999999998 2023-10-20T03:35:31.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.297748+0000", 2023-10-20T03:35:31.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7722000000000001e-05 2023-10-20T03:35:31.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.770 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.771 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:31.771 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.329666+0000", 2023-10-20T03:35:31.771 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.4143954660000002, 2023-10-20T03:35:31.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026825213000000001, 2023-10-20T03:35:31.771 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.771 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.772 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329666+0000", 2023-10-20T03:35:31.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329666+0000", 2023-10-20T03:35:31.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329667+0000", 2023-10-20T03:35:31.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2023-10-20T03:35:31.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330048+0000", 2023-10-20T03:35:31.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038081299999999997 2023-10-20T03:35:31.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330082+0000", 2023-10-20T03:35:31.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4402e-05 2023-10-20T03:35:31.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330100+0000", 2023-10-20T03:35:31.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7683e-05 2023-10-20T03:35:31.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330268+0000", 2023-10-20T03:35:31.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016784499999999999 2023-10-20T03:35:31.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330296+0000", 2023-10-20T03:35:31.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8367000000000001e-05 2023-10-20T03:35:31.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356426+0000", 2023-10-20T03:35:31.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026129432000000001 2023-10-20T03:35:31.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356492+0000", 2023-10-20T03:35:31.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6063999999999996e-05 2023-10-20T03:35:31.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.781 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.782 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 43/31 ECSubWrite(tid=81, reqid=client.4570.0:307, at_version=43'15, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:31.782 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.343216+0000", 2023-10-20T03:35:31.782 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.4008455780000002, 2023-10-20T03:35:31.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016101977, 2023-10-20T03:35:31.783 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.783 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.783 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343216+0000", 2023-10-20T03:35:31.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343216+0000", 2023-10-20T03:35:31.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343219+0000", 2023-10-20T03:35:31.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1739999999999999e-06 2023-10-20T03:35:31.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343585+0000", 2023-10-20T03:35:31.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036599000000000001 2023-10-20T03:35:31.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343611+0000", 2023-10-20T03:35:31.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6749000000000001e-05 2023-10-20T03:35:31.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343627+0000", 2023-10-20T03:35:31.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5282999999999999e-05 2023-10-20T03:35:31.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.344698+0000", 2023-10-20T03:35:31.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010717910000000001 2023-10-20T03:35:31.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.344728+0000", 2023-10-20T03:35:31.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9332e-05 2023-10-20T03:35:31.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359214+0000", 2023-10-20T03:35:31.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014485924000000001 2023-10-20T03:35:31.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359318+0000", 2023-10-20T03:35:31.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104734 2023-10-20T03:35:31.793 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.793 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.793 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.793 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s1 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:35:31.794 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.357167+0000", 2023-10-20T03:35:31.794 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.386894581, 2023-10-20T03:35:31.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025761703, 2023-10-20T03:35:31.794 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.794 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.795 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357167+0000", 2023-10-20T03:35:31.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357167+0000", 2023-10-20T03:35:31.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357169+0000", 2023-10-20T03:35:31.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.967e-06 2023-10-20T03:35:31.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357532+0000", 2023-10-20T03:35:31.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036283700000000002 2023-10-20T03:35:31.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357564+0000", 2023-10-20T03:35:31.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1798999999999998e-05 2023-10-20T03:35:31.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357615+0000", 2023-10-20T03:35:31.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1004999999999997e-05 2023-10-20T03:35:31.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357832+0000", 2023-10-20T03:35:31.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000216608 2023-10-20T03:35:31.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357857+0000", 2023-10-20T03:35:31.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5092999999999998e-05 2023-10-20T03:35:31.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.382875+0000", 2023-10-20T03:35:31.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025017978999999999 2023-10-20T03:35:31.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.382929+0000", 2023-10-20T03:35:31.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4415e-05 2023-10-20T03:35:31.804 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.804 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.804 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.804 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s1 43/31 ECSubWrite(tid=85, reqid=client.4570.0:312, at_version=43'25, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:35:31.805 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.372510+0000", 2023-10-20T03:35:31.805 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.3715519069999997, 2023-10-20T03:35:31.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011372392, 2023-10-20T03:35:31.805 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.805 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.806 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.372510+0000", 2023-10-20T03:35:31.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.372510+0000", 2023-10-20T03:35:31.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.372512+0000", 2023-10-20T03:35:31.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.643e-06 2023-10-20T03:35:31.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.372788+0000", 2023-10-20T03:35:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0002767 2023-10-20T03:35:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.372849+0000", 2023-10-20T03:35:31.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0331000000000002e-05 2023-10-20T03:35:31.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.372875+0000", 2023-10-20T03:35:31.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6250000000000001e-05 2023-10-20T03:35:31.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.374495+0000", 2023-10-20T03:35:31.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016203089999999999 2023-10-20T03:35:31.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.374521+0000", 2023-10-20T03:35:31.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5823e-05 2023-10-20T03:35:31.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.383819+0000", 2023-10-20T03:35:31.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092977019999999997 2023-10-20T03:35:31.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.383882+0000", 2023-10-20T03:35:31.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3633999999999994e-05 2023-10-20T03:35:31.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.815 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.816 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s1 43/31 ECSubWrite(tid=88, reqid=client.4570.0:313, at_version=43'26, trim_to=0'0, roll_forward_to=43'24))", 2023-10-20T03:35:31.816 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.387896+0000", 2023-10-20T03:35:31.816 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.356166322, 2023-10-20T03:35:31.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079638279999999992, 2023-10-20T03:35:31.817 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.817 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.817 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.387896+0000", 2023-10-20T03:35:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.387896+0000", 2023-10-20T03:35:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.387897+0000", 2023-10-20T03:35:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5e-06 2023-10-20T03:35:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.388093+0000", 2023-10-20T03:35:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019606800000000001 2023-10-20T03:35:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.388125+0000", 2023-10-20T03:35:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1313000000000003e-05 2023-10-20T03:35:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.388139+0000", 2023-10-20T03:35:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4802e-05 2023-10-20T03:35:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.388307+0000", 2023-10-20T03:35:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016734000000000001 2023-10-20T03:35:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.388336+0000", 2023-10-20T03:35:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9759000000000001e-05 2023-10-20T03:35:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.395764+0000", 2023-10-20T03:35:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074274249999999997 2023-10-20T03:35:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.395859+0000", 2023-10-20T03:35:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5620999999999998e-05 2023-10-20T03:35:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:320 3.2s0 3:474cd252:::smithi17039300-42:head [create,setxattr c[X]j^UsOa`q]sYrSaa]raqNm]iSd^X]OmelObeN (242) in=40b,setxattr gopkVbealbs^Z^^`^sXjNlPPYp^\\fbk\\fUnmTq[R (208) in=40b,setxattr eUZoNnss (4560) in=8b,setxattr SYo`knsRbgPQPPbXW[[ZsTehoXbSPY_WQl]_URc` (105) in=40b,setxattr \\PYbW]PkbTb^sYhsihYgqj]pROo]\\NWqNhT_lWgn (17753) in=40b,setxattr ]R_fObdq\\NUZ]QiO]j`djkVVp`bfRdph]Zbea]gg (17332) in=40b,setxattr ]ptcajatsVQWlWeoNNlPngWealknZSiQarONqbaV (1473) in=40b,setxattr `i_qfTQWSto[gYWTmYsOtOhsW[fnhaVacqaadrU] (1191) in=40b,setxattr nNhpa\\e^VfUjte_Pc[kSor^[gpr[eaflXRs]SWZr (16140) in=40b,setxattr aRNZ_mZpadm^l[rZntn`UX^RN\\cZ_TT]rRTjibjc (1121) in=40b,setxattr aVipjRkr_^aolT[aa`UQhsfb\\eRQ[ZZleb_OQOlX (2384) in=40b,setxattr m^VcgrkhPRmZlXiSggOaoPTi^]^WnQh\\aXaekmoc (947) in=40b,setxattr NikdcN_ilNkWgOQgZqrYPT\\b[drPRq\\niPPQdbVU (5260) in=40b,setxattr ajnWlVTfP^PVRs[VchU^kT_gSmcjmZRh]ofVgTrp (1437) in=40b,setxattr liRPRT`aa[cqbZkpaWa\\q[NZWUXYk[a[cicigTsV (1017) in=40b,setxattr XkR_Nkkpdeiacga[]Y^NismNTd`tQP[eb[qbrrd^ (8912) in=40b,setxattr ]mai`f_WjqaYertRarRZ[mrmWf_`mktRW`slld^_ (243) in=40b,setxattr jacQPeR\\bNj_nOTOc^WkPrSpceTds[[eee`SndWY (1086) in=40b,setxattr \\lcQhm[Vm\\jW\\cPdSlWoZcU[d^htQTrhRak`t\\hk (4068) in=40b,setxattr e`Od\\\\V]jhaQVknTVdWZOTq[eYbmlehRf^ZtoZnQ (384) in=40b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.333521+0000", 2023-10-20T03:35:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.4105404479999999, 2023-10-20T03:35:31.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016194731, 2023-10-20T03:35:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:31.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 320 2023-10-20T03:35:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333521+0000", 2023-10-20T03:35:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333521+0000", 2023-10-20T03:35:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333524+0000", 2023-10-20T03:35:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8270000000000002e-06 2023-10-20T03:35:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333727+0000", 2023-10-20T03:35:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000202504 2023-10-20T03:35:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333762+0000", 2023-10-20T03:35:31.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5271999999999997e-05 2023-10-20T03:35:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333780+0000", 2023-10-20T03:35:31.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8280000000000001e-05 2023-10-20T03:35:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333968+0000", 2023-10-20T03:35:31.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000187655 2023-10-20T03:35:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2023-10-20T03:35:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333984+0000", 2023-10-20T03:35:31.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5739e-05 2023-10-20T03:35:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.339320+0000", 2023-10-20T03:35:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053365089999999997 2023-10-20T03:35:31.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.339685+0000", 2023-10-20T03:35:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036516599999999999 2023-10-20T03:35:31.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.342206+0000", 2023-10-20T03:35:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025201440000000002 2023-10-20T03:35:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.346469+0000", 2023-10-20T03:35:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042632420000000004 2023-10-20T03:35:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:35:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.349492+0000", 2023-10-20T03:35:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030232560000000002 2023-10-20T03:35:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.349716+0000", 2023-10-20T03:35:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022413700000000001 2023-10-20T03:35:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:387 3.2s0 3:4f9c85d4:::smithi17039300-39:d [read 0~1622016,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:35:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.381906+0000", 2023-10-20T03:35:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.36215590800000003, 2023-10-20T03:35:31.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016689273000000001, 2023-10-20T03:35:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:35:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:31.845 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 387 2023-10-20T03:35:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381906+0000", 2023-10-20T03:35:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381906+0000", 2023-10-20T03:35:31.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381909+0000", 2023-10-20T03:35:31.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.965e-06 2023-10-20T03:35:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381925+0000", 2023-10-20T03:35:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5914999999999999e-05 2023-10-20T03:35:31.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381953+0000", 2023-10-20T03:35:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8430999999999999e-05 2023-10-20T03:35:31.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381968+0000", 2023-10-20T03:35:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4430000000000001e-05 2023-10-20T03:35:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382204+0000", 2023-10-20T03:35:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023644700000000001 2023-10-20T03:35:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2023-10-20T03:35:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382224+0000", 2023-10-20T03:35:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9406e-05 2023-10-20T03:35:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.387100+0000", 2023-10-20T03:35:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048765429999999997 2023-10-20T03:35:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.387279+0000", 2023-10-20T03:35:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017912599999999999 2023-10-20T03:35:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.398595+0000", 2023-10-20T03:35:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01131601 2023-10-20T03:35:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:388 3.2s0 3:4f9c85d4:::smithi17039300-39:d [read 0~1622016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:35:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.382005+0000", 2023-10-20T03:35:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.362057289, 2023-10-20T03:35:31.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019238755, 2023-10-20T03:35:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:35:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:31.858 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 388 2023-10-20T03:35:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382005+0000", 2023-10-20T03:35:31.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382005+0000", 2023-10-20T03:35:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382005+0000", 2023-10-20T03:35:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4200000000000005e-07 2023-10-20T03:35:31.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382011+0000", 2023-10-20T03:35:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0839999999999999e-06 2023-10-20T03:35:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382020+0000", 2023-10-20T03:35:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8709999999999992e-06 2023-10-20T03:35:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382029+0000", 2023-10-20T03:35:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0450000000000006e-06 2023-10-20T03:35:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382364+0000", 2023-10-20T03:35:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000334493 2023-10-20T03:35:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382374+0000", 2023-10-20T03:35:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.028e-05 2023-10-20T03:35:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.388188+0000", 2023-10-20T03:35:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058138490000000003 2023-10-20T03:35:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.388305+0000", 2023-10-20T03:35:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116747 2023-10-20T03:35:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.401243+0000", 2023-10-20T03:35:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012938544 2023-10-20T03:35:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:389 3.2s0 3:4f9c85d4:::smithi17039300-39:d [read 0~1622016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:35:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.382042+0000", 2023-10-20T03:35:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.36202032899999997, 2023-10-20T03:35:31.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021763228999999999, 2023-10-20T03:35:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:35:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:31.871 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 389 2023-10-20T03:35:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382042+0000", 2023-10-20T03:35:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382042+0000", 2023-10-20T03:35:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382042+0000", 2023-10-20T03:35:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2023-10-20T03:35:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382046+0000", 2023-10-20T03:35:31.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0319999999999997e-06 2023-10-20T03:35:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382061+0000", 2023-10-20T03:35:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4800999999999999e-05 2023-10-20T03:35:31.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382067+0000", 2023-10-20T03:35:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0630000000000002e-06 2023-10-20T03:35:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382430+0000", 2023-10-20T03:35:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000362505 2023-10-20T03:35:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382439+0000", 2023-10-20T03:35:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2199999999999998e-06 2023-10-20T03:35:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.388649+0000", 2023-10-20T03:35:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0062098459999999998 2023-10-20T03:35:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:31.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.388756+0000", 2023-10-20T03:35:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107316 2023-10-20T03:35:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.403805+0000", 2023-10-20T03:35:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015048857000000001 2023-10-20T03:35:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpReadReply(3.2s0 49/31 ECSubReadReply(tid=31, attrs_read=0))", 2023-10-20T03:35:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.389387+0000", 2023-10-20T03:35:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.35467489099999999, 2023-10-20T03:35:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066799989999999998, 2023-10-20T03:35:31.883 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.389387+0000", 2023-10-20T03:35:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.389387+0000", 2023-10-20T03:35:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.389388+0000", 2023-10-20T03:35:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.26e-06 2023-10-20T03:35:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.392200+0000", 2023-10-20T03:35:31.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028121000000000001 2023-10-20T03:35:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.392707+0000", 2023-10-20T03:35:31.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050636799999999999 2023-10-20T03:35:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.392728+0000", 2023-10-20T03:35:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0945999999999999e-05 2023-10-20T03:35:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.395965+0000", 2023-10-20T03:35:31.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032374330000000001 2023-10-20T03:35:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.396067+0000", 2023-10-20T03:35:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010189200000000001 2023-10-20T03:35:31.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.892 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpReadReply(3.2s0 49/31 ECSubReadReply(tid=31, attrs_read=0))", 2023-10-20T03:35:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.390581+0000", 2023-10-20T03:35:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.35348136699999999, 2023-10-20T03:35:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0081657050000000005, 2023-10-20T03:35:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.893 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.390581+0000", 2023-10-20T03:35:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.390581+0000", 2023-10-20T03:35:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.390582+0000", 2023-10-20T03:35:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5600000000000001e-06 2023-10-20T03:35:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.394217+0000", 2023-10-20T03:35:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036345129999999998 2023-10-20T03:35:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.394244+0000", 2023-10-20T03:35:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7223e-05 2023-10-20T03:35:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.394261+0000", 2023-10-20T03:35:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6878999999999999e-05 2023-10-20T03:35:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.396357+0000", 2023-10-20T03:35:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020962950000000002 2023-10-20T03:35:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.398746+0000", 2023-10-20T03:35:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002389235 2023-10-20T03:35:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpReadReply(3.2s0 49/31 ECSubReadReply(tid=32, attrs_read=0))", 2023-10-20T03:35:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.392770+0000", 2023-10-20T03:35:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.351292245, 2023-10-20T03:35:31.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061729849999999998, 2023-10-20T03:35:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.392770+0000", 2023-10-20T03:35:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.392770+0000", 2023-10-20T03:35:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.392771+0000", 2023-10-20T03:35:31.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1170000000000001e-06 2023-10-20T03:35:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.393767+0000", 2023-10-20T03:35:31.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00099642900000000006 2023-10-20T03:35:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.394703+0000", 2023-10-20T03:35:31.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000936173 2023-10-20T03:35:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.394721+0000", 2023-10-20T03:35:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7932999999999999e-05 2023-10-20T03:35:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.398827+0000", 2023-10-20T03:35:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041055320000000003 2023-10-20T03:35:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.398943+0000", 2023-10-20T03:35:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115801 2023-10-20T03:35:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpReadReply(3.2s0 49/31 ECSubReadReply(tid=32, attrs_read=0))", 2023-10-20T03:35:31.912 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.394293+0000", 2023-10-20T03:35:31.912 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.34976875899999998, 2023-10-20T03:35:31.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070631189999999996, 2023-10-20T03:35:31.912 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.912 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.912 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.394293+0000", 2023-10-20T03:35:31.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.394293+0000", 2023-10-20T03:35:31.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.394294+0000", 2023-10-20T03:35:31.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.167e-06 2023-10-20T03:35:31.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.396207+0000", 2023-10-20T03:35:31.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001912277 2023-10-20T03:35:31.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.396219+0000", 2023-10-20T03:35:31.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2138e-05 2023-10-20T03:35:31.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.396229+0000", 2023-10-20T03:35:31.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0353999999999999e-05 2023-10-20T03:35:31.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.399205+0000", 2023-10-20T03:35:31.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029756990000000001 2023-10-20T03:35:31.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.401356+0000", 2023-10-20T03:35:31.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021514839999999999 2023-10-20T03:35:31.920 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.920 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.920 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.921 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpReadReply(3.2s0 49/31 ECSubReadReply(tid=33, attrs_read=0))", 2023-10-20T03:35:31.921 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.396809+0000", 2023-10-20T03:35:31.921 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.34725298399999999, 2023-10-20T03:35:31.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0078625520000000001, 2023-10-20T03:35:31.922 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.922 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.922 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.396809+0000", 2023-10-20T03:35:31.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.396809+0000", 2023-10-20T03:35:31.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.396809+0000", 2023-10-20T03:35:31.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4799999999999999e-07 2023-10-20T03:35:31.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.396986+0000", 2023-10-20T03:35:31.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176763 2023-10-20T03:35:31.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.396998+0000", 2023-10-20T03:35:31.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2082000000000001e-05 2023-10-20T03:35:31.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397005+0000", 2023-10-20T03:35:31.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1069999999999999e-06 2023-10-20T03:35:31.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.401519+0000", 2023-10-20T03:35:31.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00451353 2023-10-20T03:35:31.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.404672+0000", 2023-10-20T03:35:31.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031526219999999999 2023-10-20T03:35:31.930 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.930 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.930 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.931 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 49/31 ECSubWriteReply(tid=34, last_complete=49'19, committed=1, applied=1))", 2023-10-20T03:35:31.931 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.397031+0000", 2023-10-20T03:35:31.931 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.34703139100000002, 2023-10-20T03:35:31.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070314180000000002, 2023-10-20T03:35:31.931 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.931 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.932 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397031+0000", 2023-10-20T03:35:31.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397031+0000", 2023-10-20T03:35:31.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397031+0000", 2023-10-20T03:35:31.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6300000000000001e-07 2023-10-20T03:35:31.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397033+0000", 2023-10-20T03:35:31.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1009999999999999e-06 2023-10-20T03:35:31.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397040+0000", 2023-10-20T03:35:31.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9750000000000001e-06 2023-10-20T03:35:31.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397046+0000", 2023-10-20T03:35:31.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9499999999999998e-06 2023-10-20T03:35:31.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.404000+0000", 2023-10-20T03:35:31.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069538569999999999 2023-10-20T03:35:31.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.404062+0000", 2023-10-20T03:35:31.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2172e-05 2023-10-20T03:35:31.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.939 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.940 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 49/31 ECSubWriteReply(tid=34, last_complete=49'19, committed=1, applied=1))", 2023-10-20T03:35:31.940 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.397124+0000", 2023-10-20T03:35:31.941 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.346938421, 2023-10-20T03:35:31.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070719609999999999, 2023-10-20T03:35:31.941 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.941 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.941 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397124+0000", 2023-10-20T03:35:31.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397124+0000", 2023-10-20T03:35:31.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397124+0000", 2023-10-20T03:35:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8299999999999997e-07 2023-10-20T03:35:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397126+0000", 2023-10-20T03:35:31.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.159e-06 2023-10-20T03:35:31.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397144+0000", 2023-10-20T03:35:31.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7455000000000001e-05 2023-10-20T03:35:31.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397158+0000", 2023-10-20T03:35:31.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4217e-05 2023-10-20T03:35:31.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.404144+0000", 2023-10-20T03:35:31.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069860850000000004 2023-10-20T03:35:31.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.404195+0000", 2023-10-20T03:35:31.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1561999999999999e-05 2023-10-20T03:35:31.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.949 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.950 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 48/31 ECSubWriteReply(tid=34, last_complete=49'19, committed=1, applied=1))", 2023-10-20T03:35:31.950 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.397565+0000", 2023-10-20T03:35:31.950 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.34649727499999999, 2023-10-20T03:35:31.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067417950000000001, 2023-10-20T03:35:31.950 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.951 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397565+0000", 2023-10-20T03:35:31.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397565+0000", 2023-10-20T03:35:31.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397566+0000", 2023-10-20T03:35:31.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2023-10-20T03:35:31.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397571+0000", 2023-10-20T03:35:31.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6359999999999999e-06 2023-10-20T03:35:31.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397587+0000", 2023-10-20T03:35:31.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6283e-05 2023-10-20T03:35:31.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.397595+0000", 2023-10-20T03:35:31.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0479999999999999e-06 2023-10-20T03:35:31.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.404256+0000", 2023-10-20T03:35:31.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0066606670000000003 2023-10-20T03:35:31.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.404306+0000", 2023-10-20T03:35:31.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321000000000002e-05 2023-10-20T03:35:31.958 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.958 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.958 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.959 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 49/31 ECSubWriteReply(tid=34, last_complete=49'19, committed=1, applied=1))", 2023-10-20T03:35:31.959 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.399376+0000", 2023-10-20T03:35:31.959 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.34468595400000002, 2023-10-20T03:35:31.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050376589999999999, 2023-10-20T03:35:31.959 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:31.959 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:31.960 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:31.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:31.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.399376+0000", 2023-10-20T03:35:31.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:31.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.399376+0000", 2023-10-20T03:35:31.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:31.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:31.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.399377+0000", 2023-10-20T03:35:31.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1999999999999999e-07 2023-10-20T03:35:31.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:31.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.399381+0000", 2023-10-20T03:35:31.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3830000000000002e-06 2023-10-20T03:35:31.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:31.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.399391+0000", 2023-10-20T03:35:31.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5340000000000005e-06 2023-10-20T03:35:31.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:31.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.399396+0000", 2023-10-20T03:35:31.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3890000000000004e-06 2023-10-20T03:35:31.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:31.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.404367+0000", 2023-10-20T03:35:31.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0049705870000000003 2023-10-20T03:35:31.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:31.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:31.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:31.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.404414+0000", 2023-10-20T03:35:31.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7145999999999999e-05 2023-10-20T03:35:31.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.967 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:31.967 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:31.968 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:31.968 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:35:31.969 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:31.966+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:31.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-10-20T03:35:32.070 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:32.069+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:32.100 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:32.100 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:32.100 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2023-10-20T03:35:32.101 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:32.114 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-10-20T03:35:32.126 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:32.126 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.147 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:32.147 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.170 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:32.170 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.171 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:32.169+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:32.190 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:32.191 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.213 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:32.214 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.233 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:32.233 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.233 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:32.238 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:32.238 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:32.253 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-10-20T03:35:32.254 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:32.255 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.274 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:32.275 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.298 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:32.298 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.303 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:32.302+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:32.319 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:32.319 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.337 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:32.338 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.354 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:32.354 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.372 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:32.370+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:32.376 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:32.376 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.378 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:32.378 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:32.382 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2023-10-20T03:35:32.389 INFO:tasks.rados.rados.0.smithi170.stdout:164: done (2 left) 2023-10-20T03:35:32.389 INFO:tasks.rados.rados.0.smithi170.stdout:165: snap_remove snap 5 2023-10-20T03:35:32.393 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-10-20T03:35:32.400 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:32.400 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.401 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:35:32.397+0000 7f4ce0ca8640 0 -- 172.21.15.33:0/3522895735 >> [v2:172.21.15.170:6819/826332016,v1:172.21.15.170:6822/826332016] conn(0x7f4cdc221210 msgr2=0x7f4cdc223600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-10-20T03:35:32.479 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:32.478+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:32.566 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:32.567 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:32.575 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:32.568+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:32.584 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-10-20T03:35:32.623 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:32.624 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:35:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:32.628 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:32.675 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:32.673+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:32.747 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:32.747 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:32.761 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-10-20T03:35:32.811 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:32.810+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:32.877 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:32.874+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:32.881 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:32.881 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:32.896 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-10-20T03:35:32.983 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:32.982+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:33.063 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:33.063 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:33.077 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-10-20T03:35:33.119 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:33.118+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:33.180 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:33.178+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:33.239 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:33.239 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:33.253 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-10-20T03:35:33.281 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:33.277+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:33.382 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:33.378+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:33.393 INFO:tasks.rados.rados.0.smithi170.stdout:165: done (2 left) 2023-10-20T03:35:33.393 INFO:tasks.rados.rados.0.smithi170.stdout:166: rollback oid 2 current snap is 11 2023-10-20T03:35:33.393 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 2 to 1 2023-10-20T03:35:33.394 INFO:tasks.rados.rados.0.smithi170.stdout:161: expect (ObjNum 58 snap 0 seq_num 58) 2023-10-20T03:35:33.438 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:33.438 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:33.458 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-10-20T03:35:33.475 INFO:tasks.rados.rados.0.smithi170.stdout:162: finishing copy_from racing read to smithi17039300-49 2023-10-20T03:35:33.475 INFO:tasks.rados.rados.0.smithi170.stdout:162: finishing copy_from to smithi17039300-49 2023-10-20T03:35:33.475 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 49 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-10-20T03:35:33.475 INFO:tasks.rados.rados.0.smithi170.stdout:166: finishing rollback tid 0 to smithi17039300-2 2023-10-20T03:35:33.475 INFO:tasks.rados.rados.0.smithi170.stdout:166: finishing rollback tid 1 to smithi17039300-2 2023-10-20T03:35:33.476 INFO:tasks.rados.rados.0.smithi170.stdout:166: finishing rollback tid 2 to smithi17039300-2 2023-10-20T03:35:33.476 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 27 (ObjNum 1 snap 0 seq_num 1) dirty exists 2023-10-20T03:35:33.476 INFO:tasks.rados.rados.0.smithi170.stdout:161: done (2 left) 2023-10-20T03:35:33.476 INFO:tasks.rados.rados.0.smithi170.stdout:162: done (1 left) 2023-10-20T03:35:33.476 INFO:tasks.rados.rados.0.smithi170.stdout:166: done (0 left) 2023-10-20T03:35:33.476 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 2 current snap is 11 2023-10-20T03:35:33.477 INFO:tasks.rados.rados.0.smithi170.stdout:167: seq_num 88 ranges {32768=720896,753664=442368,1196032=327680} 2023-10-20T03:35:33.480 INFO:tasks.rados.rados.0.smithi170.stdout:167: writing smithi17039300-2 from 32768 to 753664 tid 1 2023-10-20T03:35:33.483 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:33.480+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:33.484 INFO:tasks.rados.rados.0.smithi170.stdout:167: writing smithi17039300-2 from 753664 to 1196032 tid 2 2023-10-20T03:35:33.488 INFO:tasks.rados.rados.0.smithi170.stdout:167: writing smithi17039300-2 from 1196032 to 1523712 tid 3 2023-10-20T03:35:33.488 INFO:tasks.rados.rados.0.smithi170.stdout:168: delete oid 6 current snap is 11 2023-10-20T03:35:33.495 INFO:tasks.rados.rados.0.smithi170.stdout:168: done (1 left) 2023-10-20T03:35:33.495 INFO:tasks.rados.rados.0.smithi170.stdout:169: snap_remove snap 10 2023-10-20T03:35:33.583 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:33.582+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:33.599 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:33.599 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:33.613 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-10-20T03:35:33.685 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:33.681+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:33.771 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:33.771 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:33.785 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-10-20T03:35:33.785 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:33.784+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:33.886 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:33.880+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:33.907 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:33.907 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:33.921 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-10-20T03:35:33.987 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:33.984+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:34.055 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:34.055 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:34.069 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-10-20T03:35:34.087 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:34.086+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:34.188 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:34.184+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:34.193 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:34.193 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:34.208 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-10-20T03:35:34.289 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:34.286+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:34.334 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:34.334 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:34.348 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-10-20T03:35:34.382 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [6] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 10, 14, 3, 7, 11, 15] 2023-10-20T03:35:34.382 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-10-20T03:35:34.382 INFO:tasks.thrashosds.thrasher:Removing osd 12, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2023-10-20T03:35:34.383 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 12 2023-10-20T03:35:34.390 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:34.384+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:34.404 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-10-20T03:35:34.405 INFO:tasks.rados.rados.0.smithi170.stdout:169: done (1 left) 2023-10-20T03:35:34.405 INFO:tasks.rados.rados.0.smithi170.stdout:170: snap_remove snap 7 2023-10-20T03:35:34.475 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:34.475 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:34.491 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-10-20T03:35:34.491 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:34.488+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:34.591 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:34.589+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:34.629 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:35:34.650 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:34.650 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:34.666 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-10-20T03:35:34.693 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:34.690+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:34.738 INFO:tasks.ceph.mon.a.smithi033.stderr:2023-10-20T03:35:34.737+0000 7f44d76b2640 -1 mon.a@0(leader).osd e52 definitely_dead 0 2023-10-20T03:35:34.771 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:34.771 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:35:34.792 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:34.793 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:35:34.793 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:34.792+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:34.813 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:34.814 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:35:34.831 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:34.831 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:34.835 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:34.835 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:35:34.845 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-10-20T03:35:34.853 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:34.854 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:35:34.875 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:34.876 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:35:34.876 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:34.894 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:34.890+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:34.896 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:34.897 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:35:34.914 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:34.914 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:35:34.936 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:34.936 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-10-20T03:35:34.957 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:34.957 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:35:34.971 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:34.971 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:34.971 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:34.972 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:34.977 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:34.978 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:35:34.986 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-10-20T03:35:34.995 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:34.995 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:35:34.996 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:34.993+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:35.016 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:35.016 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:35:35.037 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:35.037 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:35:35.058 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:35.058 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:35:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:35.061 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:35.096 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:35.096+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:35.114 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:35.114 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:35.114 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:35:35.114 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:35:35.115 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:35.128 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-10-20T03:35:35.197 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:35.196+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:35.250 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:35.251 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:35.251 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:35.251 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:35:35.251 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.252 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=21, reqid=client.4570.0:198, at_version=36'13, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:35.252 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:35.252 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 16.980863027000002, 2023-10-20T03:35:35.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086318609999999994, 2023-10-20T03:35:35.252 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.253 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:35.253 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.253 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:35.254 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.254 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.254 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.254 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.254 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:35.255 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.255 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.255 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.255 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.255 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:35.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.58e-07 2023-10-20T03:35:35.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.256 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.267902+0000", 2023-10-20T03:35:35.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014514409999999999 2023-10-20T03:35:35.257 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.257 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.257 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.257 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.267916+0000", 2023-10-20T03:35:35.257 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4664999999999999e-05 2023-10-20T03:35:35.257 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.258 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.258 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.258 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.267931+0000", 2023-10-20T03:35:35.258 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4192999999999999e-05 2023-10-20T03:35:35.259 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.259 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.259 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.259 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.268007+0000", 2023-10-20T03:35:35.259 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6058999999999998e-05 2023-10-20T03:35:35.259 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.260 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.260 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.260 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.268017+0000", 2023-10-20T03:35:35.260 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0648999999999999e-05 2023-10-20T03:35:35.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.261 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.261 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.261 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275053+0000", 2023-10-20T03:35:35.261 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0070355699999999997 2023-10-20T03:35:35.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.262 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.262 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.262 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275082+0000", 2023-10-20T03:35:35.262 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8926000000000001e-05 2023-10-20T03:35:35.262 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.263 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.263 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.263 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.263 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.263 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=22, reqid=client.4570.0:199, at_version=36'14, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:35.264 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.284006+0000", 2023-10-20T03:35:35.264 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 16.963306580000001, 2023-10-20T03:35:35.264 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0061959429999999998, 2023-10-20T03:35:35.264 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.264 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:35.265 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.265 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.265 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.265 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284006+0000", 2023-10-20T03:35:35.265 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.266 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.266 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284006+0000", 2023-10-20T03:35:35.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284008+0000", 2023-10-20T03:35:35.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.398e-06 2023-10-20T03:35:35.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.269 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284090+0000", 2023-10-20T03:35:35.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2686000000000001e-05 2023-10-20T03:35:35.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.270 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284111+0000", 2023-10-20T03:35:35.270 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0363000000000001e-05 2023-10-20T03:35:35.270 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.270 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.270 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.271 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284123+0000", 2023-10-20T03:35:35.271 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2213e-05 2023-10-20T03:35:35.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.271 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.272 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284222+0000", 2023-10-20T03:35:35.272 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8587999999999999e-05 2023-10-20T03:35:35.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.272 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.272 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284237+0000", 2023-10-20T03:35:35.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.541e-05 2023-10-20T03:35:35.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.290159+0000", 2023-10-20T03:35:35.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0059216299999999998 2023-10-20T03:35:35.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.275 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.275 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.290202+0000", 2023-10-20T03:35:35.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3655000000000001e-05 2023-10-20T03:35:35.275 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.275 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.276 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.276 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:212 3.5s0 3:aed1dba0:::smithi17039300-29:2 [read 0~425984,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:35.276 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.277328+0000", 2023-10-20T03:35:35.277 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.969985429999999, 2023-10-20T03:35:35.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085440599999999992, 2023-10-20T03:35:35.277 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.277 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:35.277 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:35.278 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:35.278 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:35.278 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 212 2023-10-20T03:35:35.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.278 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277328+0000", 2023-10-20T03:35:35.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277328+0000", 2023-10-20T03:35:35.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.281 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277330+0000", 2023-10-20T03:35:35.281 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7690000000000001e-06 2023-10-20T03:35:35.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.281 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.281 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277343+0000", 2023-10-20T03:35:35.282 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2221e-05 2023-10-20T03:35:35.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.282 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.282 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.282 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277368+0000", 2023-10-20T03:35:35.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5544000000000001e-05 2023-10-20T03:35:35.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.283 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.283 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277382+0000", 2023-10-20T03:35:35.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4246e-05 2023-10-20T03:35:35.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277510+0000", 2023-10-20T03:35:35.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012724599999999999 2023-10-20T03:35:35.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2023-10-20T03:35:35.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277524+0000", 2023-10-20T03:35:35.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4466e-05 2023-10-20T03:35:35.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.281643+0000", 2023-10-20T03:35:35.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041191680000000003 2023-10-20T03:35:35.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.287 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:35.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.281815+0000", 2023-10-20T03:35:35.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000171991 2023-10-20T03:35:35.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.288 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.288 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.288 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.285872+0000", 2023-10-20T03:35:35.288 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0040564090000000004 2023-10-20T03:35:35.288 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.288 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.289 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.289 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.289 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:213 3.5s0 3:aed1dba0:::smithi17039300-29:2 [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:35.289 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.277413+0000", 2023-10-20T03:35:35.290 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.969899558, 2023-10-20T03:35:35.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0087379769999999992, 2023-10-20T03:35:35.290 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.290 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:35.291 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:35.291 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:35.291 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:35.291 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 213 2023-10-20T03:35:35.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.292 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277413+0000", 2023-10-20T03:35:35.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277413+0000", 2023-10-20T03:35:35.294 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.294 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.294 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.294 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.295 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277414+0000", 2023-10-20T03:35:35.295 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7599999999999997e-07 2023-10-20T03:35:35.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.296 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277418+0000", 2023-10-20T03:35:35.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3599999999999998e-06 2023-10-20T03:35:35.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277428+0000", 2023-10-20T03:35:35.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0034000000000001e-05 2023-10-20T03:35:35.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.300 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277435+0000", 2023-10-20T03:35:35.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2049999999999999e-06 2023-10-20T03:35:35.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.300 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.301 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277660+0000", 2023-10-20T03:35:35.301 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022565199999999999 2023-10-20T03:35:35.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.301 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:35.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277673+0000", 2023-10-20T03:35:35.302 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2330000000000001e-05 2023-10-20T03:35:35.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.282724+0000", 2023-10-20T03:35:35.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0050513370000000004 2023-10-20T03:35:35.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:35.303 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.282891+0000", 2023-10-20T03:35:35.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016650500000000001 2023-10-20T03:35:35.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.286151+0000", 2023-10-20T03:35:35.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003260878 2023-10-20T03:35:35.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.305 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.306 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:214 3.5s0 3:aed1dba0:::smithi17039300-29:2 [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:35.306 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.277483+0000", 2023-10-20T03:35:35.306 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.96983017, 2023-10-20T03:35:35.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.009248025, 2023-10-20T03:35:35.306 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.307 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:35.307 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:35.307 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:35.307 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:35.307 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 214 2023-10-20T03:35:35.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.308 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277483+0000", 2023-10-20T03:35:35.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277483+0000", 2023-10-20T03:35:35.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277483+0000", 2023-10-20T03:35:35.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.75e-07 2023-10-20T03:35:35.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277489+0000", 2023-10-20T03:35:35.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.57e-06 2023-10-20T03:35:35.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.312 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.312 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277497+0000", 2023-10-20T03:35:35.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4419999999999998e-06 2023-10-20T03:35:35.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.313 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.313 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277504+0000", 2023-10-20T03:35:35.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3430000000000001e-06 2023-10-20T03:35:35.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277741+0000", 2023-10-20T03:35:35.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00023721699999999999 2023-10-20T03:35:35.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:35.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277751+0000", 2023-10-20T03:35:35.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0122999999999999e-05 2023-10-20T03:35:35.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283390+0000", 2023-10-20T03:35:35.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056394009999999996 2023-10-20T03:35:35.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:35.317 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283539+0000", 2023-10-20T03:35:35.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014805499999999999 2023-10-20T03:35:35.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.286731+0000", 2023-10-20T03:35:35.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003192299 2023-10-20T03:35:35.318 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.318 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.319 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.319 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.319 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:316 3.5s0 3:aed1dba0:::smithi17039300-29:head [append 0~0] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:35.319 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.364850+0000", 2023-10-20T03:35:35.319 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.8824633899999998, 2023-10-20T03:35:35.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016187792999999999, 2023-10-20T03:35:35.320 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.320 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:35.320 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:35.320 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:35.321 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:35.321 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 316 2023-10-20T03:35:35.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.321 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364850+0000", 2023-10-20T03:35:35.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364850+0000", 2023-10-20T03:35:35.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.324 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364852+0000", 2023-10-20T03:35:35.324 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0260000000000002e-06 2023-10-20T03:35:35.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.325 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364861+0000", 2023-10-20T03:35:35.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5899999999999997e-06 2023-10-20T03:35:35.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.325 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.326 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364880+0000", 2023-10-20T03:35:35.326 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9245999999999998e-05 2023-10-20T03:35:35.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.327 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364892+0000", 2023-10-20T03:35:35.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.184e-05 2023-10-20T03:35:35.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.328 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.328 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364987+0000", 2023-10-20T03:35:35.328 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4752000000000004e-05 2023-10-20T03:35:35.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2023-10-20T03:35:35.329 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364997+0000", 2023-10-20T03:35:35.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0319999999999999e-05 2023-10-20T03:35:35.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.368108+0000", 2023-10-20T03:35:35.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0031111200000000002 2023-10-20T03:35:35.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:35.331 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.368196+0000", 2023-10-20T03:35:35.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7773000000000001e-05 2023-10-20T03:35:35.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.368815+0000", 2023-10-20T03:35:35.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00061905900000000004 2023-10-20T03:35:35.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370813+0000", 2023-10-20T03:35:35.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019981780000000002 2023-10-20T03:35:35.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:35.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.380943+0000", 2023-10-20T03:35:35.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010129487 2023-10-20T03:35:35.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.334 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.381037+0000", 2023-10-20T03:35:35.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4401999999999995e-05 2023-10-20T03:35:35.335 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.335 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.335 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.336 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:317 3.5s0 3:aed1dba0:::smithi17039300-29:head [rollback 9] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:35.336 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.364924+0000", 2023-10-20T03:35:35.336 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.8823885320000002, 2023-10-20T03:35:35.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017874489, 2023-10-20T03:35:35.337 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.337 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:35.337 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:35.337 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:35.337 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:35.337 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 317 2023-10-20T03:35:35.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.338 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.338 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.338 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364924+0000", 2023-10-20T03:35:35.338 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.339 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.339 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364924+0000", 2023-10-20T03:35:35.339 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.340 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.340 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364925+0000", 2023-10-20T03:35:35.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8599999999999998e-07 2023-10-20T03:35:35.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.341 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364929+0000", 2023-10-20T03:35:35.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7309999999999999e-06 2023-10-20T03:35:35.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.342 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364940+0000", 2023-10-20T03:35:35.342 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1236999999999999e-05 2023-10-20T03:35:35.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.342 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.343 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364947+0000", 2023-10-20T03:35:35.343 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0469999999999996e-06 2023-10-20T03:35:35.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.343 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.343 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365101+0000", 2023-10-20T03:35:35.344 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015391 2023-10-20T03:35:35.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.344 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.344 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:35.344 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365110+0000", 2023-10-20T03:35:35.344 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2e-06 2023-10-20T03:35:35.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.345 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.345 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.345 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.369303+0000", 2023-10-20T03:35:35.345 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041924149999999997 2023-10-20T03:35:35.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.346 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:35.346 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.369380+0000", 2023-10-20T03:35:35.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.716e-05 2023-10-20T03:35:35.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.347 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.369806+0000", 2023-10-20T03:35:35.347 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00042586099999999998 2023-10-20T03:35:35.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.348 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.348 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.372757+0000", 2023-10-20T03:35:35.348 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029512430000000001 2023-10-20T03:35:35.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.348 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.349 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:35.349 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.382729+0000", 2023-10-20T03:35:35.349 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099724180000000003 2023-10-20T03:35:35.349 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.349 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.350 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.350 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.382799+0000", 2023-10-20T03:35:35.350 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9680999999999999e-05 2023-10-20T03:35:35.350 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.350 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.350 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.351 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.351 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:318 3.5s0 3:aed1dba0:::smithi17039300-29:head [append 0~0] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:35.351 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:35:35.351 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.8823283790000005, 2023-10-20T03:35:35.352 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018045379, 2023-10-20T03:35:35.352 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.352 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:35.352 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:35.352 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:35.352 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:35.353 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 318 2023-10-20T03:35:35.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.353 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.354 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:35:35.354 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:35:35.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.356 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:35:35.356 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8899999999999999e-07 2023-10-20T03:35:35.356 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.356 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.356 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.357 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364989+0000", 2023-10-20T03:35:35.357 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7079999999999999e-06 2023-10-20T03:35:35.357 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.357 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.357 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.358 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364998+0000", 2023-10-20T03:35:35.358 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2229999999999996e-06 2023-10-20T03:35:35.358 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.358 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.358 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.359 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365004+0000", 2023-10-20T03:35:35.359 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1999999999999999e-06 2023-10-20T03:35:35.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.359 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.360 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.360 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365157+0000", 2023-10-20T03:35:35.360 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015222799999999999 2023-10-20T03:35:35.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.360 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:35.361 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365166+0000", 2023-10-20T03:35:35.361 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6739999999999992e-06 2023-10-20T03:35:35.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.361 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.361 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.362 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370187+0000", 2023-10-20T03:35:35.362 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0050205279999999998 2023-10-20T03:35:35.362 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.362 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.362 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:35.362 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370256+0000", 2023-10-20T03:35:35.363 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9151000000000003e-05 2023-10-20T03:35:35.363 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.363 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.363 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.363 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370618+0000", 2023-10-20T03:35:35.363 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00036223100000000002 2023-10-20T03:35:35.364 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.364 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.364 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.364 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.375514+0000", 2023-10-20T03:35:35.364 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0048961220000000001 2023-10-20T03:35:35.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.365 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:35.365 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.382954+0000", 2023-10-20T03:35:35.365 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074393280000000003 2023-10-20T03:35:35.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.366 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.366 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.383030+0000", 2023-10-20T03:35:35.366 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6396999999999994e-05 2023-10-20T03:35:35.366 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.367 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.367 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.367 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:35.367 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:35:35.368 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8586472710000006, 2023-10-20T03:35:35.368 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028147078999999998, 2023-10-20T03:35:35.368 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.368 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:35.368 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.369 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.369 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.369 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:35:35.369 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.370 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.370 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:35:35.370 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.371 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.371 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:35:35.371 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4600000000000004e-07 2023-10-20T03:35:35.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.372 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.372 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388785+0000", 2023-10-20T03:35:35.372 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000118604 2023-10-20T03:35:35.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388827+0000", 2023-10-20T03:35:35.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2404000000000003e-05 2023-10-20T03:35:35.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.374 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.374 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388843+0000", 2023-10-20T03:35:35.374 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5894000000000001e-05 2023-10-20T03:35:35.374 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.374 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.374 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.375 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.389002+0000", 2023-10-20T03:35:35.375 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000158383 2023-10-20T03:35:35.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.375 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.375 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.375 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.389019+0000", 2023-10-20T03:35:35.376 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7266e-05 2023-10-20T03:35:35.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.376 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.376 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416738+0000", 2023-10-20T03:35:35.376 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027718779999999998 2023-10-20T03:35:35.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.377 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.377 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416813+0000", 2023-10-20T03:35:35.377 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5102e-05 2023-10-20T03:35:35.378 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.378 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.378 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.379 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:340 3.5s0 3:aed1dba0:::smithi17039300-29:head [create,setxattr nWX_d[W\\dXUWOgbXndrSbcTU`mgjR_]o\\nNcXpin (1403) in=40b,setxattr TSP[ddnm_WR]nh]QhSXpkpmmWeOP^b]baemT^Uqr (630) in=40b,setxattr R[QU^NWYrjfarmhlWmZlUkP\\ipWan`SnqZOX`VWh (806) in=40b,setxattr ]hV\\^[PQZO]n^abhilRfldoUhgtl\\\\oel`mgahd_ (3368) in=40b,setxattr OYc`XQ\\VjZfQ`]OsbemUo_mfVUSc^pNo\\[\\ldrhg (1087) in=40b,setxattr `pZtTmno_eqZOONfatgZ_`nnhemaaQWiteO]dc[V (175) in=40b,setxattr PdQa`feQapkrnjfOUYUeqsOt_hrsg\\noqrcV]pSh (981) in=40b,setxattr bk_QpaSsdlQfT`oa[pTQS_[ad]isfYm^tQNdhSiW (1718) in=40b,setxattr Ys`Y_UZ^PXhpgS`VS]`eVgkcmT^QW[aWscssRRSr (10033) in=40b,setxattr [jd]Q\\qemWTVcc`hiZnsrqrggetajZRa]\\_llS`h (1367) in=40b,setxattr `UajoeZh]tRSf\\[nnVat^TjN^_q`ooQ_br`itgZm (244) in=40b,setxattr Tng[WcWa_dsnRXtZNa^`fle`gQ___mar\\fgRsZVY (665) in=40b,setxattr nogmhepVZfQtnZc^cZSeaaogjklZkrff_Utg`UR^ (15856) in=40b,setxattr gfqSsQY[][rrac[gmaQrTSacNWttTTXhanmRa^fa (1998) in=40b,setxattr V`Y[ddb`_dfep[_]oUoTqQj\\UW[gkesnlgNPYtPa (1438) in=40b,setxattr YgoRVqUOj_mXPXsq^PdjaaolmQsTsTsZpaddktl[ (1400) in=40b,setxattr smn[QVPVkfVs]oTip]jtirOtU\\Pa\\Ym^poZgfZaq (1263) in=40b,setxattr TQSmSs^Z]\\rdosO_a^TOU^[p`jkQb`dXXh]s`Stq (909) in=40b,setxattr faPhpqVXmWh`VeNZo[b^[kYl^[[rYYPRrk_[ON`e (14664) in=40b,setxattr ksO_UmqqQefe[Q\\njt[W_roWeei`]Sn^TV]O^iSW (1963) in=40b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:35.379 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.397436+0000", 2023-10-20T03:35:35.379 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8498765749999997, 2023-10-20T03:35:35.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024477860000000001, 2023-10-20T03:35:35.379 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.379 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:35.380 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:35.380 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:35.380 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:35.380 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 340 2023-10-20T03:35:35.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.380 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.381 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.381 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.381 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397436+0000", 2023-10-20T03:35:35.381 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.381 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.382 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.382 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.382 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397436+0000", 2023-10-20T03:35:35.382 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.382 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.383 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.383 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.383 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397439+0000", 2023-10-20T03:35:35.383 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6929999999999999e-06 2023-10-20T03:35:35.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.384 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.384 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.384 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397608+0000", 2023-10-20T03:35:35.384 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016918 2023-10-20T03:35:35.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.384 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.385 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.385 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397637+0000", 2023-10-20T03:35:35.385 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9057e-05 2023-10-20T03:35:35.385 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.385 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.386 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.386 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397653+0000", 2023-10-20T03:35:35.386 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5430999999999998e-05 2023-10-20T03:35:35.386 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.386 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.386 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.387 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397749+0000", 2023-10-20T03:35:35.387 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6694000000000006e-05 2023-10-20T03:35:35.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.387 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:35.388 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397877+0000", 2023-10-20T03:35:35.388 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012762599999999999 2023-10-20T03:35:35.388 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.388 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.388 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.389 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.399009+0000", 2023-10-20T03:35:35.389 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011323640000000001 2023-10-20T03:35:35.389 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.389 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.389 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.389 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416407+0000", 2023-10-20T03:35:35.390 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017398034999999999 2023-10-20T03:35:35.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.390 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.390 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:35.390 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.421802+0000", 2023-10-20T03:35:35.391 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053944320000000002 2023-10-20T03:35:35.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.391 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.391 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.421914+0000", 2023-10-20T03:35:35.391 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000112348 2023-10-20T03:35:35.392 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.392 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.392 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.392 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.393 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:35.393 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.401616+0000", 2023-10-20T03:35:35.393 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8456968200000006, 2023-10-20T03:35:35.393 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015282946, 2023-10-20T03:35:35.393 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.394 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:35.394 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.394 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.394 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.394 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401616+0000", 2023-10-20T03:35:35.395 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.395 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.395 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.395 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.395 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401616+0000", 2023-10-20T03:35:35.396 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.396 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.396 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.396 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401617+0000", 2023-10-20T03:35:35.396 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.105e-06 2023-10-20T03:35:35.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.397 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.397 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.397 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401748+0000", 2023-10-20T03:35:35.397 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013050599999999999 2023-10-20T03:35:35.398 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.398 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.398 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.398 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401765+0000", 2023-10-20T03:35:35.398 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7450999999999999e-05 2023-10-20T03:35:35.399 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.399 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.399 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.399 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401774+0000", 2023-10-20T03:35:35.399 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9760000000000001e-06 2023-10-20T03:35:35.399 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.400 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.400 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.400 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404428+0000", 2023-10-20T03:35:35.400 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00265409 2023-10-20T03:35:35.400 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.401 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.401 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.401 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404442+0000", 2023-10-20T03:35:35.401 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3602e-05 2023-10-20T03:35:35.401 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.401 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.402 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.402 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416847+0000", 2023-10-20T03:35:35.402 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012404837 2023-10-20T03:35:35.402 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.402 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.403 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.403 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416899+0000", 2023-10-20T03:35:35.403 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2379000000000001e-05 2023-10-20T03:35:35.403 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.403 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.404 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.404 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.404 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.404 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=53, reqid=client.4570.0:335, at_version=44'20, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:35.404 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.403071+0000", 2023-10-20T03:35:35.405 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8442418200000006, 2023-10-20T03:35:35.405 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014226947, 2023-10-20T03:35:35.405 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.405 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:35.405 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.405 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.406 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.406 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403071+0000", 2023-10-20T03:35:35.406 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.406 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.407 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.407 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403071+0000", 2023-10-20T03:35:35.407 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.407 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.407 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.407 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.408 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403072+0000", 2023-10-20T03:35:35.408 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2299999999999999e-07 2023-10-20T03:35:35.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.408 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.408 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.408 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403107+0000", 2023-10-20T03:35:35.409 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4900000000000001e-05 2023-10-20T03:35:35.409 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.409 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.409 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.409 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403125+0000", 2023-10-20T03:35:35.410 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8386999999999999e-05 2023-10-20T03:35:35.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.410 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.410 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.410 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403138+0000", 2023-10-20T03:35:35.410 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2979999999999999e-05 2023-10-20T03:35:35.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.411 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.411 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.411 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413715+0000", 2023-10-20T03:35:35.411 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010576538999999999 2023-10-20T03:35:35.412 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.412 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.412 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.412 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413733+0000", 2023-10-20T03:35:35.412 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8357000000000001e-05 2023-10-20T03:35:35.412 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.413 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.413 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.413 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417237+0000", 2023-10-20T03:35:35.413 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0035037689999999999 2023-10-20T03:35:35.413 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.414 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.414 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.414 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417298+0000", 2023-10-20T03:35:35.414 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1092000000000004e-05 2023-10-20T03:35:35.414 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.415 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.415 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.415 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.415 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.415 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 44/31 ECSubWriteReply(tid=22, last_complete=44'12, committed=1, applied=1))", 2023-10-20T03:35:35.416 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.404745+0000", 2023-10-20T03:35:35.416 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8425683680000002, 2023-10-20T03:35:35.416 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089739039999999996, 2023-10-20T03:35:35.416 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.416 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:35.416 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.417 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.417 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.417 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404745+0000", 2023-10-20T03:35:35.417 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.417 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.418 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.418 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.418 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404745+0000", 2023-10-20T03:35:35.418 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.418 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.419 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.419 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.419 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404746+0000", 2023-10-20T03:35:35.419 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.156e-06 2023-10-20T03:35:35.419 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.420 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.420 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.420 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404753+0000", 2023-10-20T03:35:35.420 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7009999999999996e-06 2023-10-20T03:35:35.420 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.420 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.421 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.421 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404774+0000", 2023-10-20T03:35:35.421 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0390999999999998e-05 2023-10-20T03:35:35.421 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.421 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.422 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.422 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404790+0000", 2023-10-20T03:35:35.422 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5744e-05 2023-10-20T03:35:35.422 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.422 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.423 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.423 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413701+0000", 2023-10-20T03:35:35.423 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089117050000000007 2023-10-20T03:35:35.423 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.423 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.424 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.424 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413719+0000", 2023-10-20T03:35:35.424 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7207e-05 2023-10-20T03:35:35.424 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.424 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.424 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.425 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.425 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.425 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=54, reqid=client.4570.0:336, at_version=44'21, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:35.425 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.405096+0000", 2023-10-20T03:35:35.425 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8422167629999997, 2023-10-20T03:35:35.426 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012879895000000001, 2023-10-20T03:35:35.426 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.426 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:35.426 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.426 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.427 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.427 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405096+0000", 2023-10-20T03:35:35.427 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.427 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.427 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.428 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.428 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405096+0000", 2023-10-20T03:35:35.428 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.428 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.428 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.428 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.429 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405097+0000", 2023-10-20T03:35:35.429 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.035e-06 2023-10-20T03:35:35.429 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.429 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.429 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405121+0000", 2023-10-20T03:35:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4097e-05 2023-10-20T03:35:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405177+0000", 2023-10-20T03:35:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5220000000000003e-05 2023-10-20T03:35:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405192+0000", 2023-10-20T03:35:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.579e-05 2023-10-20T03:35:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416447+0000", 2023-10-20T03:35:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011254188 2023-10-20T03:35:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416463+0000", 2023-10-20T03:35:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6441000000000001e-05 2023-10-20T03:35:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417913+0000", 2023-10-20T03:35:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014495809999999999 2023-10-20T03:35:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417976+0000", 2023-10-20T03:35:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3542999999999998e-05 2023-10-20T03:35:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 44/31 ECSubWriteReply(tid=22, last_complete=44'12, committed=1, applied=1))", 2023-10-20T03:35:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.406456+0000", 2023-10-20T03:35:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8408569149999998, 2023-10-20T03:35:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010387031999999999, 2023-10-20T03:35:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406456+0000", 2023-10-20T03:35:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406456+0000", 2023-10-20T03:35:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406457+0000", 2023-10-20T03:35:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6199999999999996e-07 2023-10-20T03:35:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406462+0000", 2023-10-20T03:35:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2889999999999997e-06 2023-10-20T03:35:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406479+0000", 2023-10-20T03:35:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6963e-05 2023-10-20T03:35:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406488+0000", 2023-10-20T03:35:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6200000000000005e-06 2023-10-20T03:35:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416476+0000", 2023-10-20T03:35:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099880660000000003 2023-10-20T03:35:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416843+0000", 2023-10-20T03:35:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000367232 2023-10-20T03:35:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 44/31 ECSubWriteReply(tid=22, last_complete=44'12, committed=1, applied=1))", 2023-10-20T03:35:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.406529+0000", 2023-10-20T03:35:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8407836609999997, 2023-10-20T03:35:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010245637, 2023-10-20T03:35:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406529+0000", 2023-10-20T03:35:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406529+0000", 2023-10-20T03:35:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406530+0000", 2023-10-20T03:35:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9599999999999999e-07 2023-10-20T03:35:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406541+0000", 2023-10-20T03:35:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1208e-05 2023-10-20T03:35:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406557+0000", 2023-10-20T03:35:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5563999999999999e-05 2023-10-20T03:35:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406566+0000", 2023-10-20T03:35:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0769999999999997e-06 2023-10-20T03:35:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416751+0000", 2023-10-20T03:35:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010184794 2023-10-20T03:35:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416775+0000", 2023-10-20T03:35:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4298e-05 2023-10-20T03:35:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:35:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.393394+0000", 2023-10-20T03:35:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.8539184730000002, 2023-10-20T03:35:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039046538999999998, 2023-10-20T03:35:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393394+0000", 2023-10-20T03:35:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393394+0000", 2023-10-20T03:35:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393395+0000", 2023-10-20T03:35:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.16e-07 2023-10-20T03:35:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394217+0000", 2023-10-20T03:35:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00082179599999999996 2023-10-20T03:35:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394258+0000", 2023-10-20T03:35:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0546000000000001e-05 2023-10-20T03:35:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394283+0000", 2023-10-20T03:35:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5707e-05 2023-10-20T03:35:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394593+0000", 2023-10-20T03:35:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031000800000000002 2023-10-20T03:35:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394630+0000", 2023-10-20T03:35:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6885999999999999e-05 2023-10-20T03:35:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432353+0000", 2023-10-20T03:35:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037723055999999998 2023-10-20T03:35:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432441+0000", 2023-10-20T03:35:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7724000000000002e-05 2023-10-20T03:35:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:392 3.5s0 3:aed1dba0:::smithi17039300-29:head [stat,delete] snapc f=[f,d,b,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:35:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.382023+0000", 2023-10-20T03:35:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.865290012, 2023-10-20T03:35:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019348521, 2023-10-20T03:35:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 392 2023-10-20T03:35:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382023+0000", 2023-10-20T03:35:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382023+0000", 2023-10-20T03:35:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382026+0000", 2023-10-20T03:35:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3629999999999998e-06 2023-10-20T03:35:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382040+0000", 2023-10-20T03:35:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3913e-05 2023-10-20T03:35:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382075+0000", 2023-10-20T03:35:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4678999999999998e-05 2023-10-20T03:35:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382093+0000", 2023-10-20T03:35:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7779999999999999e-05 2023-10-20T03:35:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382244+0000", 2023-10-20T03:35:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015178300000000001 2023-10-20T03:35:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2023-10-20T03:35:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382257+0000", 2023-10-20T03:35:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.288e-05 2023-10-20T03:35:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.387421+0000", 2023-10-20T03:35:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0051631560000000003 2023-10-20T03:35:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.387619+0000", 2023-10-20T03:35:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000198712 2023-10-20T03:35:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.389154+0000", 2023-10-20T03:35:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001535193 2023-10-20T03:35:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.393045+0000", 2023-10-20T03:35:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0038909639999999998 2023-10-20T03:35:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.401188+0000", 2023-10-20T03:35:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081430219999999998 2023-10-20T03:35:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.401371+0000", 2023-10-20T03:35:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018307600000000001 2023-10-20T03:35:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 49/31 ECSubWrite(tid=62, reqid=osd.12.0:62, at_version=49'25, trim_to=0'0, roll_forward_to=48'23))", 2023-10-20T03:35:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.409244+0000", 2023-10-20T03:35:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.8380685589999999, 2023-10-20T03:35:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044812430000000002, 2023-10-20T03:35:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409244+0000", 2023-10-20T03:35:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409244+0000", 2023-10-20T03:35:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409245+0000", 2023-10-20T03:35:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1e-07 2023-10-20T03:35:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409274+0000", 2023-10-20T03:35:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9135000000000001e-05 2023-10-20T03:35:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409306+0000", 2023-10-20T03:35:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2536000000000002e-05 2023-10-20T03:35:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409321+0000", 2023-10-20T03:35:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.451e-05 2023-10-20T03:35:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409394+0000", 2023-10-20T03:35:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2648e-05 2023-10-20T03:35:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409405+0000", 2023-10-20T03:35:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1409e-05 2023-10-20T03:35:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.413677+0000", 2023-10-20T03:35:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042716189999999999 2023-10-20T03:35:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.413726+0000", 2023-10-20T03:35:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.9076000000000002e-05 2023-10-20T03:35:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 49/31 ECSubWrite(tid=63, reqid=osd.12.0:63, at_version=49'27, trim_to=0'0, roll_forward_to=48'23))", 2023-10-20T03:35:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.409342+0000", 2023-10-20T03:35:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.8379708290000001, 2023-10-20T03:35:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0054493629999999996, 2023-10-20T03:35:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409342+0000", 2023-10-20T03:35:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409342+0000", 2023-10-20T03:35:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409343+0000", 2023-10-20T03:35:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3700000000000001e-07 2023-10-20T03:35:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409374+0000", 2023-10-20T03:35:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1304999999999999e-05 2023-10-20T03:35:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409387+0000", 2023-10-20T03:35:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2757e-05 2023-10-20T03:35:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409396+0000", 2023-10-20T03:35:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5000000000000005e-06 2023-10-20T03:35:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409895+0000", 2023-10-20T03:35:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00049927999999999995 2023-10-20T03:35:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409910+0000", 2023-10-20T03:35:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4151000000000001e-05 2023-10-20T03:35:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414725+0000", 2023-10-20T03:35:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0048157110000000003 2023-10-20T03:35:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414792+0000", 2023-10-20T03:35:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6222000000000004e-05 2023-10-20T03:35:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:35.505 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:35.507 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-10-20T03:35:35.508 INFO:teuthology.orchestra.run.smithi033.stderr:marked out osd.12. 2023-10-20T03:35:35.508 INFO:tasks.rados.rados.0.smithi170.stdout:170: done (1 left) 2023-10-20T03:35:35.509 INFO:tasks.rados.rados.0.smithi170.stdout:171: copy_from oid 8 from oid 20 current snap is 11 2023-10-20T03:35:35.509 INFO:tasks.rados.rados.0.smithi170.stdout:167: finishing write tid 1 to smithi17039300-2 2023-10-20T03:35:35.509 INFO:tasks.rados.rados.0.smithi170.stdout:167: finishing write tid 2 to smithi17039300-2 2023-10-20T03:35:35.509 INFO:tasks.rados.rados.0.smithi170.stdout:167: finishing write tid 3 to smithi17039300-2 2023-10-20T03:35:35.509 INFO:tasks.rados.rados.0.smithi170.stdout:167: finishing write tid 4 to smithi17039300-2 2023-10-20T03:35:35.510 INFO:tasks.rados.rados.0.smithi170.stdout:167: finishing write tid 5 to smithi17039300-2 2023-10-20T03:35:35.510 INFO:tasks.rados.rados.0.smithi170.stdout:167: oid 2 updating version 0 to 30 2023-10-20T03:35:35.510 INFO:tasks.rados.rados.0.smithi170.stdout:167: oid 2 updating version 30 to 31 2023-10-20T03:35:35.510 INFO:tasks.rados.rados.0.smithi170.stdout:167: oid 2 version 31 is already newer than 28 2023-10-20T03:35:35.510 INFO:tasks.rados.rados.0.smithi170.stdout:167: oid 2 version 31 is already newer than 29 2023-10-20T03:35:35.511 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 31 (ObjNum 88 snap 11 seq_num 88) dirty exists 2023-10-20T03:35:35.511 INFO:tasks.rados.rados.0.smithi170.stdout:167: left oid 2 (ObjNum 88 snap 11 seq_num 88) 2023-10-20T03:35:35.511 INFO:tasks.rados.rados.0.smithi170.stdout:167: done (1 left) 2023-10-20T03:35:35.511 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 26 current snap is 11 2023-10-20T03:35:35.511 INFO:tasks.rados.rados.0.smithi170.stdout:172: seq_num 90 ranges {425984=753664,1179648=720896,1900544=49152} 2023-10-20T03:35:35.512 INFO:tasks.rados.rados.0.smithi170.stdout:172: writing smithi17039300-26 from 425984 to 1179648 tid 1 2023-10-20T03:35:35.512 INFO:tasks.rados.rados.0.smithi170.stdout:172: writing smithi17039300-26 from 1179648 to 1900544 tid 2 2023-10-20T03:35:35.512 INFO:tasks.rados.rados.0.smithi170.stdout:172: writing smithi17039300-26 from 1900544 to 1949696 tid 3 2023-10-20T03:35:35.512 INFO:tasks.rados.rados.0.smithi170.stdout:173: snap_remove snap 9 2023-10-20T03:35:35.513 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2023-10-20T03:35:35.513 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:35.500+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:35.607 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:35.604+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:35.682 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:35.683 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:35.697 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-10-20T03:35:35.708 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:35.704+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:35.809 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:35.805+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:35.818 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:35:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:35:35.819 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:35.832 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-10-20T03:35:35.918 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:35.917+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:35.998 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:35:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:35:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:35:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.638766235, 2023-10-20T03:35:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010383547999999999, 2023-10-20T03:35:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:35:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:35:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356840+0000", 2023-10-20T03:35:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1999999999999999e-07 2023-10-20T03:35:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357506+0000", 2023-10-20T03:35:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00066620500000000001 2023-10-20T03:35:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357522+0000", 2023-10-20T03:35:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5964000000000002e-05 2023-10-20T03:35:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357535+0000", 2023-10-20T03:35:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3046e-05 2023-10-20T03:35:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357625+0000", 2023-10-20T03:35:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9691999999999994e-05 2023-10-20T03:35:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357636+0000", 2023-10-20T03:35:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1202e-05 2023-10-20T03:35:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.367176+0000", 2023-10-20T03:35:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0095396060000000008 2023-10-20T03:35:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.367223+0000", 2023-10-20T03:35:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7113000000000001e-05 2023-10-20T03:35:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=85, reqid=client.4570.0:312, at_version=43'25, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:35:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:35:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.621253836999999, 2023-10-20T03:35:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0052771249999999997, 2023-10-20T03:35:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:35:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:35:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374353+0000", 2023-10-20T03:35:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6199999999999999e-06 2023-10-20T03:35:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374496+0000", 2023-10-20T03:35:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014282800000000001 2023-10-20T03:35:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374515+0000", 2023-10-20T03:35:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8281e-05 2023-10-20T03:35:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374527+0000", 2023-10-20T03:35:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2835e-05 2023-10-20T03:35:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374607+0000", 2023-10-20T03:35:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9283999999999995e-05 2023-10-20T03:35:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374617+0000", 2023-10-20T03:35:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0394000000000001e-05 2023-10-20T03:35:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.379594+0000", 2023-10-20T03:35:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049767650000000002 2023-10-20T03:35:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.379629+0000", 2023-10-20T03:35:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5117999999999998e-05 2023-10-20T03:35:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=88, reqid=client.4570.0:313, at_version=43'26, trim_to=0'0, roll_forward_to=43'24))", 2023-10-20T03:35:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:35:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 10.608063534999999, 2023-10-20T03:35:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049208230000000004, 2023-10-20T03:35:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:35:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:35:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387544+0000", 2023-10-20T03:35:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6109999999999999e-06 2023-10-20T03:35:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387771+0000", 2023-10-20T03:35:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022766 2023-10-20T03:35:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387788+0000", 2023-10-20T03:35:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7079999999999999e-05 2023-10-20T03:35:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387799+0000", 2023-10-20T03:35:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0608000000000001e-05 2023-10-20T03:35:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387881+0000", 2023-10-20T03:35:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2060999999999999e-05 2023-10-20T03:35:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387892+0000", 2023-10-20T03:35:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0912e-05 2023-10-20T03:35:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.392420+0000", 2023-10-20T03:35:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0045283320000000004 2023-10-20T03:35:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.392463+0000", 2023-10-20T03:35:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2558999999999997e-05 2023-10-20T03:35:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:333 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~606208] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:35:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.6096757630000003, 2023-10-20T03:35:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036441440999999998, 2023-10-20T03:35:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 333 2023-10-20T03:35:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:35:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:35:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385932+0000", 2023-10-20T03:35:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9929999999999998e-06 2023-10-20T03:35:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386889+0000", 2023-10-20T03:35:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000957437 2023-10-20T03:35:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386904+0000", 2023-10-20T03:35:36.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4669e-05 2023-10-20T03:35:36.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.039 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.039 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386910+0000", 2023-10-20T03:35:36.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3040000000000001e-06 2023-10-20T03:35:36.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.040 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386972+0000", 2023-10-20T03:35:36.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2003999999999998e-05 2023-10-20T03:35:36.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.041 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:36.041 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387043+0000", 2023-10-20T03:35:36.041 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0591000000000002e-05 2023-10-20T03:35:36.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388080+0000", 2023-10-20T03:35:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010373190000000001 2023-10-20T03:35:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393723+0000", 2023-10-20T03:35:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056426899999999997 2023-10-20T03:35:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422308+0000", 2023-10-20T03:35:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028584822999999999 2023-10-20T03:35:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422371+0000", 2023-10-20T03:35:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3610999999999998e-05 2023-10-20T03:35:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:334 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~720896] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:35:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.6025413640000004, 2023-10-20T03:35:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029571147999999998, 2023-10-20T03:35:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 334 2023-10-20T03:35:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:35:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:35:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393066+0000", 2023-10-20T03:35:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8819999999999999e-06 2023-10-20T03:35:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393827+0000", 2023-10-20T03:35:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000760844 2023-10-20T03:35:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393842+0000", 2023-10-20T03:35:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5432999999999999e-05 2023-10-20T03:35:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393850+0000", 2023-10-20T03:35:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0199999999999994e-06 2023-10-20T03:35:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393915+0000", 2023-10-20T03:35:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4123999999999998e-05 2023-10-20T03:35:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394021+0000", 2023-10-20T03:35:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010668299999999999 2023-10-20T03:35:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394832+0000", 2023-10-20T03:35:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081066099999999996 2023-10-20T03:35:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405535+0000", 2023-10-20T03:35:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010703351 2023-10-20T03:35:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422553+0000", 2023-10-20T03:35:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017018021000000001 2023-10-20T03:35:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422635+0000", 2023-10-20T03:35:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2128999999999999e-05 2023-10-20T03:35:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:335 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~49152] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:35:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.6017343880000006, 2023-10-20T03:35:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029832813999999999, 2023-10-20T03:35:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 335 2023-10-20T03:35:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:35:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:35:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393872+0000", 2023-10-20T03:35:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6000000000000003e-07 2023-10-20T03:35:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393919+0000", 2023-10-20T03:35:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6576000000000003e-05 2023-10-20T03:35:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393928+0000", 2023-10-20T03:35:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1220000000000001e-06 2023-10-20T03:35:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393937+0000", 2023-10-20T03:35:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1740000000000006e-06 2023-10-20T03:35:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401885+0000", 2023-10-20T03:35:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079486149999999992 2023-10-20T03:35:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.402012+0000", 2023-10-20T03:35:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012668 2023-10-20T03:35:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403063+0000", 2023-10-20T03:35:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010508410000000001 2023-10-20T03:35:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405538+0000", 2023-10-20T03:35:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024754519999999999 2023-10-20T03:35:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.423630+0000", 2023-10-20T03:35:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018091716000000001 2023-10-20T03:35:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.423704+0000", 2023-10-20T03:35:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3878e-05 2023-10-20T03:35:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:336 3.ds0 3:b8ff490e:::smithi17039300-34:head [setxattr _header (56) in=7b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:35:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.6016550130000002, 2023-10-20T03:35:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030244758, 2023-10-20T03:35:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 336 2023-10-20T03:35:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:35:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:35:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:35:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8400000000000004e-07 2023-10-20T03:35:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393955+0000", 2023-10-20T03:35:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8210000000000003e-06 2023-10-20T03:35:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393963+0000", 2023-10-20T03:35:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8359999999999992e-06 2023-10-20T03:35:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393968+0000", 2023-10-20T03:35:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3689999999999998e-06 2023-10-20T03:35:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404110+0000", 2023-10-20T03:35:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010141582 2023-10-20T03:35:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404246+0000", 2023-10-20T03:35:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013631799999999999 2023-10-20T03:35:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405009+0000", 2023-10-20T03:35:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00076302800000000001 2023-10-20T03:35:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406198+0000", 2023-10-20T03:35:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011889610000000001 2023-10-20T03:35:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424049+0000", 2023-10-20T03:35:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017850957000000001 2023-10-20T03:35:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424195+0000", 2023-10-20T03:35:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014630200000000001 2023-10-20T03:35:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:337 3.ds0 3:b8ff490e:::smithi17039300-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:35:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.6016282289999992, 2023-10-20T03:35:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.032645448000000001, 2023-10-20T03:35:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 337 2023-10-20T03:35:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:35:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:35:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393978+0000", 2023-10-20T03:35:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0800000000000005e-07 2023-10-20T03:35:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393981+0000", 2023-10-20T03:35:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7199999999999998e-06 2023-10-20T03:35:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393988+0000", 2023-10-20T03:35:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5939999999999996e-06 2023-10-20T03:35:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393993+0000", 2023-10-20T03:35:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0200000000000002e-06 2023-10-20T03:35:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405399+0000", 2023-10-20T03:35:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011406229 2023-10-20T03:35:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405522+0000", 2023-10-20T03:35:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012226199999999999 2023-10-20T03:35:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424833+0000", 2023-10-20T03:35:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019311524 2023-10-20T03:35:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424976+0000", 2023-10-20T03:35:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014297600000000001 2023-10-20T03:35:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.426623+0000", 2023-10-20T03:35:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0016466149999999999 2023-10-20T03:35:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:384 3.ds0 3:ba203010:::smithi17039300-4:head [copy-from ver 21] snapc f=[f,d,b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:35:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:35:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.6248449469999997, 2023-10-20T03:35:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.077402023, 2023-10-20T03:35:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 384 2023-10-20T03:35:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:35:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:35:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370763+0000", 2023-10-20T03:35:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4569999999999999e-06 2023-10-20T03:35:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370776+0000", 2023-10-20T03:35:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2483000000000001e-05 2023-10-20T03:35:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370809+0000", 2023-10-20T03:35:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2916000000000001e-05 2023-10-20T03:35:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370826+0000", 2023-10-20T03:35:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7183000000000001e-05 2023-10-20T03:35:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371075+0000", 2023-10-20T03:35:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000249387 2023-10-20T03:35:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371389+0000", 2023-10-20T03:35:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031429899999999998 2023-10-20T03:35:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392222+0000", 2023-10-20T03:35:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020832532000000001 2023-10-20T03:35:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.447199+0000", 2023-10-20T03:35:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054976917 2023-10-20T03:35:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.448009+0000", 2023-10-20T03:35:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080978099999999998 2023-10-20T03:35:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.448163+0000", 2023-10-20T03:35:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015406799999999999 2023-10-20T03:35:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:385 3.ds0 3:ba203010:::smithi17039300-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:35:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:35:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.6247448230000003, 2023-10-20T03:35:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.074370642000000001, 2023-10-20T03:35:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 385 2023-10-20T03:35:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:35:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:35:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370862+0000", 2023-10-20T03:35:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2699999999999998e-07 2023-10-20T03:35:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370866+0000", 2023-10-20T03:35:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.42e-06 2023-10-20T03:35:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370880+0000", 2023-10-20T03:35:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3833e-05 2023-10-20T03:35:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370889+0000", 2023-10-20T03:35:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5900000000000008e-06 2023-10-20T03:35:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371887+0000", 2023-10-20T03:35:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00099814499999999998 2023-10-20T03:35:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2023-10-20T03:35:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.372069+0000", 2023-10-20T03:35:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018220899999999999 2023-10-20T03:35:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445044+0000", 2023-10-20T03:35:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072974686999999996 2023-10-20T03:35:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445137+0000", 2023-10-20T03:35:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3694000000000001e-05 2023-10-20T03:35:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445231+0000", 2023-10-20T03:35:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4136999999999997e-05 2023-10-20T03:35:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(osd.12.22:4 3.ds0 3:b8ff490e:::smithi17039300-34:head [assert-version v21,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:35:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:35:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.6222960999999998, 2023-10-20T03:35:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083526739999999992, 2023-10-20T03:35:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:35:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:35:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373312+0000", 2023-10-20T03:35:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.424e-06 2023-10-20T03:35:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373326+0000", 2023-10-20T03:35:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4226000000000001e-05 2023-10-20T03:35:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373357+0000", 2023-10-20T03:35:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0698999999999999e-05 2023-10-20T03:35:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373388+0000", 2023-10-20T03:35:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0846000000000002e-05 2023-10-20T03:35:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373551+0000", 2023-10-20T03:35:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016350300000000001 2023-10-20T03:35:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373703+0000", 2023-10-20T03:35:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015216200000000001 2023-10-20T03:35:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.381662+0000", 2023-10-20T03:35:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079588139999999998 2023-10-20T03:35:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpReadReply(3.ds0 48/31 ECSubReadReply(tid=59, attrs_read=0))", 2023-10-20T03:35:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:35:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.6193635820000001, 2023-10-20T03:35:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0055763890000000002, 2023-10-20T03:35:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:35:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:35:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376244+0000", 2023-10-20T03:35:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7570000000000001e-06 2023-10-20T03:35:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380643+0000", 2023-10-20T03:35:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00439931 2023-10-20T03:35:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380671+0000", 2023-10-20T03:35:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8019000000000001e-05 2023-10-20T03:35:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380685+0000", 2023-10-20T03:35:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4334e-05 2023-10-20T03:35:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380780+0000", 2023-10-20T03:35:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4087999999999998e-05 2023-10-20T03:35:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.381818+0000", 2023-10-20T03:35:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001038881 2023-10-20T03:35:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs2 48/47 ECSubWrite(tid=115, reqid=osd.13.0:115, at_version=48'27, trim_to=0'0, roll_forward_to=0'0))", 2023-10-20T03:35:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:35:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.6060657100000002, 2023-10-20T03:35:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047760049999999998, 2023-10-20T03:35:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:35:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:35:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389541+0000", 2023-10-20T03:35:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5659999999999999e-06 2023-10-20T03:35:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389579+0000", 2023-10-20T03:35:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7070999999999998e-05 2023-10-20T03:35:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389605+0000", 2023-10-20T03:35:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6848000000000001e-05 2023-10-20T03:35:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389626+0000", 2023-10-20T03:35:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0579000000000001e-05 2023-10-20T03:35:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389781+0000", 2023-10-20T03:35:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015502300000000001 2023-10-20T03:35:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389807+0000", 2023-10-20T03:35:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6418999999999999e-05 2023-10-20T03:35:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394222+0000", 2023-10-20T03:35:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044143139999999999 2023-10-20T03:35:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394316+0000", 2023-10-20T03:35:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4184999999999994e-05 2023-10-20T03:35:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:35:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:35:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.5661905980000004, 2023-10-20T03:35:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015918188, 2023-10-20T03:35:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:35:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:35:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429417+0000", 2023-10-20T03:35:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9429999999999999e-06 2023-10-20T03:35:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429432+0000", 2023-10-20T03:35:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4942e-05 2023-10-20T03:35:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429459+0000", 2023-10-20T03:35:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6740000000000001e-05 2023-10-20T03:35:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429472+0000", 2023-10-20T03:35:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3382e-05 2023-10-20T03:35:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445294+0000", 2023-10-20T03:35:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015821894999999999 2023-10-20T03:35:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445333+0000", 2023-10-20T03:35:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9286000000000003e-05 2023-10-20T03:35:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:35:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:35:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.5658416150000001, 2023-10-20T03:35:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01562377, 2023-10-20T03:35:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:35:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:35:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429765+0000", 2023-10-20T03:35:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0079999999999999e-06 2023-10-20T03:35:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429772+0000", 2023-10-20T03:35:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2230000000000001e-06 2023-10-20T03:35:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429793+0000", 2023-10-20T03:35:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0707999999999999e-05 2023-10-20T03:35:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429806+0000", 2023-10-20T03:35:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3528e-05 2023-10-20T03:35:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445370+0000", 2023-10-20T03:35:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015563917 2023-10-20T03:35:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445388+0000", 2023-10-20T03:35:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7385999999999999e-05 2023-10-20T03:35:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:35:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:35:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.564190719, 2023-10-20T03:35:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014045581999999999, 2023-10-20T03:35:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:35:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:35:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431417+0000", 2023-10-20T03:35:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4159999999999998e-06 2023-10-20T03:35:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431433+0000", 2023-10-20T03:35:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5653000000000001e-05 2023-10-20T03:35:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431472+0000", 2023-10-20T03:35:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9232e-05 2023-10-20T03:35:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431492+0000", 2023-10-20T03:35:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9550000000000001e-05 2023-10-20T03:35:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445435+0000", 2023-10-20T03:35:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01394307 2023-10-20T03:35:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445460+0000", 2023-10-20T03:35:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5661e-05 2023-10-20T03:35:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:35:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:35:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.5633847190000001, 2023-10-20T03:35:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013263258999999999, 2023-10-20T03:35:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:35:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:35:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432228+0000", 2023-10-20T03:35:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.08e-06 2023-10-20T03:35:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432236+0000", 2023-10-20T03:35:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6629999999999993e-06 2023-10-20T03:35:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432260+0000", 2023-10-20T03:35:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3998999999999998e-05 2023-10-20T03:35:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432272+0000", 2023-10-20T03:35:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2023e-05 2023-10-20T03:35:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445470+0000", 2023-10-20T03:35:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013198507 2023-10-20T03:35:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445484+0000", 2023-10-20T03:35:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3987e-05 2023-10-20T03:35:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:35:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:35:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.5630803699999998, 2023-10-20T03:35:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012997794, 2023-10-20T03:35:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:35:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:35:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432526+0000", 2023-10-20T03:35:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1400000000000001e-06 2023-10-20T03:35:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432532+0000", 2023-10-20T03:35:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.823e-06 2023-10-20T03:35:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432552+0000", 2023-10-20T03:35:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0032999999999999e-05 2023-10-20T03:35:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432567+0000", 2023-10-20T03:35:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4932999999999999e-05 2023-10-20T03:35:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445506+0000", 2023-10-20T03:35:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012939315 2023-10-20T03:35:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445523+0000", 2023-10-20T03:35:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6549999999999999e-05 2023-10-20T03:35:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s6 49/31 ECSubWrite(tid=25, reqid=osd.10.0:24, at_version=0'0, trim_to=0'0, roll_forward_to=49'15))", 2023-10-20T03:35:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.408039+0000", 2023-10-20T03:35:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.587566603, 2023-10-20T03:35:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0033096330000000002, 2023-10-20T03:35:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408039+0000", 2023-10-20T03:35:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408039+0000", 2023-10-20T03:35:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408044+0000", 2023-10-20T03:35:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3399999999999997e-06 2023-10-20T03:35:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408062+0000", 2023-10-20T03:35:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7615999999999999e-05 2023-10-20T03:35:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408079+0000", 2023-10-20T03:35:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7435000000000002e-05 2023-10-20T03:35:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408092+0000", 2023-10-20T03:35:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2948e-05 2023-10-20T03:35:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408206+0000", 2023-10-20T03:35:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011359299999999999 2023-10-20T03:35:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408217+0000", 2023-10-20T03:35:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0784e-05 2023-10-20T03:35:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.411294+0000", 2023-10-20T03:35:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0030770179999999999 2023-10-20T03:35:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.411349+0000", 2023-10-20T03:35:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4898999999999997e-05 2023-10-20T03:35:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs2 49/47 ECSubWrite(tid=118, reqid=osd.13.0:118, at_version=49'29, trim_to=0'0, roll_forward_to=48'27))", 2023-10-20T03:35:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:35:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.5847471390000001, 2023-10-20T03:35:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043378410000000003, 2023-10-20T03:35:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:35:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:35:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410859+0000", 2023-10-20T03:35:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0699999999999999e-07 2023-10-20T03:35:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410884+0000", 2023-10-20T03:35:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.529e-05 2023-10-20T03:35:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410898+0000", 2023-10-20T03:35:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.383e-05 2023-10-20T03:35:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410928+0000", 2023-10-20T03:35:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0329e-05 2023-10-20T03:35:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.411037+0000", 2023-10-20T03:35:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010869099999999999 2023-10-20T03:35:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.411048+0000", 2023-10-20T03:35:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0905e-05 2023-10-20T03:35:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415159+0000", 2023-10-20T03:35:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041115520000000001 2023-10-20T03:35:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415196+0000", 2023-10-20T03:35:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7036999999999998e-05 2023-10-20T03:35:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:36.227 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:36.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-10-20T03:35:36.229 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:36.225+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:36.329 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:36.329+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:36.363 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:36.363 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:36.363 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:35:36.363 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:36.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-10-20T03:35:36.424 INFO:tasks.rados.rados.0.smithi170.stdout:173: done (2 left) 2023-10-20T03:35:36.424 INFO:tasks.rados.rados.0.smithi170.stdout:174: setattr oid 5 current snap is 11 2023-10-20T03:35:36.424 INFO:tasks.rados.rados.0.smithi170.stdout:172: finishing write tid 1 to smithi17039300-26 2023-10-20T03:35:36.425 INFO:tasks.rados.rados.0.smithi170.stdout:172: finishing write tid 2 to smithi17039300-26 2023-10-20T03:35:36.425 INFO:tasks.rados.rados.0.smithi170.stdout:172: finishing write tid 3 to smithi17039300-26 2023-10-20T03:35:36.425 INFO:tasks.rados.rados.0.smithi170.stdout:172: finishing write tid 4 to smithi17039300-26 2023-10-20T03:35:36.425 INFO:tasks.rados.rados.0.smithi170.stdout:172: finishing write tid 5 to smithi17039300-26 2023-10-20T03:35:36.425 INFO:tasks.rados.rados.0.smithi170.stdout:172: oid 26 updating version 0 to 31 2023-10-20T03:35:36.426 INFO:tasks.rados.rados.0.smithi170.stdout:172: oid 26 updating version 31 to 33 2023-10-20T03:35:36.426 INFO:tasks.rados.rados.0.smithi170.stdout:172: oid 26 version 33 is already newer than 32 2023-10-20T03:35:36.426 INFO:tasks.rados.rados.0.smithi170.stdout:172: oid 26 updating version 33 to 34 2023-10-20T03:35:36.426 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 26 v 34 (ObjNum 90 snap 11 seq_num 90) dirty exists 2023-10-20T03:35:36.426 INFO:tasks.rados.rados.0.smithi170.stdout:172: left oid 26 (ObjNum 90 snap 11 seq_num 90) 2023-10-20T03:35:36.427 INFO:tasks.rados.rados.0.smithi170.stdout:172: done (2 left) 2023-10-20T03:35:36.427 INFO:tasks.rados.rados.0.smithi170.stdout:175: rollback oid 35 current snap is 11 2023-10-20T03:35:36.427 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 35 to 1 2023-10-20T03:35:36.427 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 49 current snap is 11 2023-10-20T03:35:36.428 INFO:tasks.rados.rados.0.smithi170.stdout:176: seq_num 91 ranges {802816=770048,1572864=475136,2048000=294912} 2023-10-20T03:35:36.431 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:36.426+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:36.454 INFO:tasks.rados.rados.0.smithi170.stdout:176: writing smithi17039300-49 from 802816 to 1572864 tid 1 2023-10-20T03:35:36.462 INFO:tasks.rados.rados.0.smithi170.stdout:176: writing smithi17039300-49 from 1572864 to 2048000 tid 2 2023-10-20T03:35:36.468 INFO:tasks.rados.rados.0.smithi170.stdout:176: writing smithi17039300-49 from 2048000 to 2342912 tid 3 2023-10-20T03:35:36.468 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 50 current snap is 11 2023-10-20T03:35:36.468 INFO:tasks.rados.rados.0.smithi170.stdout:177: seq_num 92 ranges {835584=786432,1622016=491520,2113536=278528} 2023-10-20T03:35:36.490 INFO:tasks.rados.rados.0.smithi170.stdout:177: writing smithi17039300-50 from 835584 to 1622016 tid 1 2023-10-20T03:35:36.495 INFO:tasks.rados.rados.0.smithi170.stdout:177: writing smithi17039300-50 from 1622016 to 2113536 tid 2 2023-10-20T03:35:36.498 INFO:tasks.rados.rados.0.smithi170.stdout:177: writing smithi17039300-50 from 2113536 to 2392064 tid 3 2023-10-20T03:35:36.499 INFO:tasks.rados.rados.0.smithi170.stdout:178: snap_create 2023-10-20T03:35:36.499 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 20 (ObjNum 63 snap 3 seq_num 63) dirty exists 2023-10-20T03:35:36.499 INFO:tasks.rados.rados.0.smithi170.stdout:176: finishing write tid 1 to smithi17039300-49 2023-10-20T03:35:36.499 INFO:tasks.rados.rados.0.smithi170.stdout:176: finishing write tid 2 to smithi17039300-49 2023-10-20T03:35:36.500 INFO:tasks.rados.rados.0.smithi170.stdout:176: finishing write tid 3 to smithi17039300-49 2023-10-20T03:35:36.502 INFO:tasks.rados.rados.0.smithi170.stdout:176: finishing write tid 4 to smithi17039300-49 2023-10-20T03:35:36.505 INFO:tasks.rados.rados.0.smithi170.stdout:176: finishing write tid 5 to smithi17039300-49 2023-10-20T03:35:36.505 INFO:tasks.rados.rados.0.smithi170.stdout:176: oid 49 updating version 0 to 35 2023-10-20T03:35:36.505 INFO:tasks.rados.rados.0.smithi170.stdout:176: oid 49 version 35 is already newer than 34 2023-10-20T03:35:36.505 INFO:tasks.rados.rados.0.smithi170.stdout:176: oid 49 version 35 is already newer than 32 2023-10-20T03:35:36.505 INFO:tasks.rados.rados.0.smithi170.stdout:176: oid 49 version 35 is already newer than 33 2023-10-20T03:35:36.506 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 49 v 35 (ObjNum 91 snap 11 seq_num 91) dirty exists 2023-10-20T03:35:36.506 INFO:tasks.rados.rados.0.smithi170.stdout:176: left oid 49 (ObjNum 91 snap 11 seq_num 91) 2023-10-20T03:35:36.518 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:36.518 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:36.518 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:35:36.518 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:35:36.519 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:36.531 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:36.528+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:36.532 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-10-20T03:35:36.632 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:36.628+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:36.706 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:36.706 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:35:36.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:35:36.706 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2023-10-20T03:35:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.bs1 41/31 ECSubWrite(tid=77, reqid=client.4570.0:293, at_version=41'10, trim_to=0'0, roll_forward_to=39'8))", 2023-10-20T03:35:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:23.461353+0000", 2023-10-20T03:35:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 13.241071883, 2023-10-20T03:35:36.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018876751000000001, 2023-10-20T03:35:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.461353+0000", 2023-10-20T03:35:36.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.461353+0000", 2023-10-20T03:35:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.461354+0000", 2023-10-20T03:35:36.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.029e-06 2023-10-20T03:35:36.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462244+0000", 2023-10-20T03:35:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089060400000000003 2023-10-20T03:35:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462291+0000", 2023-10-20T03:35:36.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6476e-05 2023-10-20T03:35:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462316+0000", 2023-10-20T03:35:36.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5320000000000002e-05 2023-10-20T03:35:36.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462615+0000", 2023-10-20T03:35:36.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00029899800000000003 2023-10-20T03:35:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462645+0000", 2023-10-20T03:35:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0352999999999999e-05 2023-10-20T03:35:36.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.480122+0000", 2023-10-20T03:35:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017476588000000001 2023-10-20T03:35:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.480229+0000", 2023-10-20T03:35:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107383 2023-10-20T03:35:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.717 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s5 43/31 ECSubWrite(tid=18, reqid=client.4570.0:316, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2023-10-20T03:35:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.368489+0000", 2023-10-20T03:35:36.718 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.333935194, 2023-10-20T03:35:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010940494, 2023-10-20T03:35:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368489+0000", 2023-10-20T03:35:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.720 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368489+0000", 2023-10-20T03:35:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368490+0000", 2023-10-20T03:35:36.722 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6700000000000003e-07 2023-10-20T03:35:36.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.722 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.722 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368554+0000", 2023-10-20T03:35:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4163000000000002e-05 2023-10-20T03:35:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.723 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368592+0000", 2023-10-20T03:35:36.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8336e-05 2023-10-20T03:35:36.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368617+0000", 2023-10-20T03:35:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5043000000000001e-05 2023-10-20T03:35:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368948+0000", 2023-10-20T03:35:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000330658 2023-10-20T03:35:36.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368976+0000", 2023-10-20T03:35:36.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7463000000000001e-05 2023-10-20T03:35:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.379338+0000", 2023-10-20T03:35:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010362227999999999 2023-10-20T03:35:36.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.379430+0000", 2023-10-20T03:35:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2035999999999998e-05 2023-10-20T03:35:36.728 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s5 43/31 ECSubWrite(tid=19, reqid=client.4570.0:317, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2023-10-20T03:35:36.729 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.369495+0000", 2023-10-20T03:35:36.730 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.332929928, 2023-10-20T03:35:36.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010000620999999999, 2023-10-20T03:35:36.730 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.730 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.730 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369495+0000", 2023-10-20T03:35:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369495+0000", 2023-10-20T03:35:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.732 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369496+0000", 2023-10-20T03:35:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.717e-06 2023-10-20T03:35:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369549+0000", 2023-10-20T03:35:36.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2416000000000001e-05 2023-10-20T03:35:36.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369581+0000", 2023-10-20T03:35:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1776999999999997e-05 2023-10-20T03:35:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369602+0000", 2023-10-20T03:35:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1713e-05 2023-10-20T03:35:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.372674+0000", 2023-10-20T03:35:36.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0030722129999999999 2023-10-20T03:35:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.372701+0000", 2023-10-20T03:35:36.737 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.656e-05 2023-10-20T03:35:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.379439+0000", 2023-10-20T03:35:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.006737799 2023-10-20T03:35:36.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.379495+0000", 2023-10-20T03:35:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6425999999999998e-05 2023-10-20T03:35:36.739 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s5 43/31 ECSubWrite(tid=20, reqid=client.4570.0:318, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2023-10-20T03:35:36.740 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.370301+0000", 2023-10-20T03:35:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.332123921000001, 2023-10-20T03:35:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010565028000000001, 2023-10-20T03:35:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.742 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.742 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370301+0000", 2023-10-20T03:35:36.742 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370301+0000", 2023-10-20T03:35:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.743 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370302+0000", 2023-10-20T03:35:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7370000000000001e-06 2023-10-20T03:35:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.744 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370346+0000", 2023-10-20T03:35:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3625999999999999e-05 2023-10-20T03:35:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.745 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370381+0000", 2023-10-20T03:35:36.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5040000000000003e-05 2023-10-20T03:35:36.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370403+0000", 2023-10-20T03:35:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2269999999999999e-05 2023-10-20T03:35:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.378341+0000", 2023-10-20T03:35:36.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0079379429999999994 2023-10-20T03:35:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.378367+0000", 2023-10-20T03:35:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6089999999999999e-05 2023-10-20T03:35:36.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.749 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.380767+0000", 2023-10-20T03:35:36.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023998520000000001 2023-10-20T03:35:36.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.380866+0000", 2023-10-20T03:35:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8469999999999997e-05 2023-10-20T03:35:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.750 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.cs5 44/31 ECSubWrite(tid=58, reqid=client.4570.0:330, at_version=44'21, trim_to=0'0, roll_forward_to=41'19))", 2023-10-20T03:35:36.751 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.371654+0000", 2023-10-20T03:35:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.330770877000001, 2023-10-20T03:35:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0046381649999999996, 2023-10-20T03:35:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.753 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.371654+0000", 2023-10-20T03:35:36.753 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.371654+0000", 2023-10-20T03:35:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.371654+0000", 2023-10-20T03:35:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9699999999999996e-07 2023-10-20T03:35:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.755 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.371711+0000", 2023-10-20T03:35:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6946e-05 2023-10-20T03:35:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.756 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.371738+0000", 2023-10-20T03:35:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6910000000000002e-05 2023-10-20T03:35:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.371760+0000", 2023-10-20T03:35:36.757 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1923e-05 2023-10-20T03:35:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.371969+0000", 2023-10-20T03:35:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000208882 2023-10-20T03:35:36.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.371997+0000", 2023-10-20T03:35:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8483000000000002e-05 2023-10-20T03:35:36.759 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.376200+0000", 2023-10-20T03:35:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0042028350000000003 2023-10-20T03:35:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.760 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.376292+0000", 2023-10-20T03:35:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1588999999999993e-05 2023-10-20T03:35:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.761 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.cs5 44/31 ECSubWrite(tid=60, reqid=client.4570.0:332, at_version=44'23, trim_to=0'0, roll_forward_to=41'19))", 2023-10-20T03:35:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.373828+0000", 2023-10-20T03:35:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.328597026000001, 2023-10-20T03:35:36.762 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0061883260000000001, 2023-10-20T03:35:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.373828+0000", 2023-10-20T03:35:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.764 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.373828+0000", 2023-10-20T03:35:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.373829+0000", 2023-10-20T03:35:36.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.756e-06 2023-10-20T03:35:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.373868+0000", 2023-10-20T03:35:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9135999999999999e-05 2023-10-20T03:35:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.373894+0000", 2023-10-20T03:35:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5907000000000001e-05 2023-10-20T03:35:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.767 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.373913+0000", 2023-10-20T03:35:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8196999999999999e-05 2023-10-20T03:35:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.375685+0000", 2023-10-20T03:35:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0017726059999999999 2023-10-20T03:35:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.769 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.375710+0000", 2023-10-20T03:35:36.770 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.529e-05 2023-10-20T03:35:36.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.770 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.770 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.770 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.379823+0000", 2023-10-20T03:35:36.770 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0041129249999999999 2023-10-20T03:35:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.380016+0000", 2023-10-20T03:35:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192509 2023-10-20T03:35:36.771 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.772 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s5 44/31 ECSubWrite(tid=22, reqid=client.4570.0:340, at_version=44'12, trim_to=0'0, roll_forward_to=43'10))", 2023-10-20T03:35:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:35:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.303515165, 2023-10-20T03:35:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0052282980000000001, 2023-10-20T03:35:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.773 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.774 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:35:36.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:35:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398910+0000", 2023-10-20T03:35:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2023-10-20T03:35:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399110+0000", 2023-10-20T03:35:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019963500000000001 2023-10-20T03:35:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399161+0000", 2023-10-20T03:35:36.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1050000000000001e-05 2023-10-20T03:35:36.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399186+0000", 2023-10-20T03:35:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4998999999999999e-05 2023-10-20T03:35:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.780 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399478+0000", 2023-10-20T03:35:36.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00029179199999999997 2023-10-20T03:35:36.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.780 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399504+0000", 2023-10-20T03:35:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6509999999999999e-05 2023-10-20T03:35:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.781 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404033+0000", 2023-10-20T03:35:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045288849999999999 2023-10-20T03:35:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.782 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404138+0000", 2023-10-20T03:35:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010465 2023-10-20T03:35:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.fs6 49/47 ECSubWrite(tid=36, reqid=osd.3.0:36, at_version=49'23, trim_to=0'0, roll_forward_to=48'21))", 2023-10-20T03:35:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.388819+0000", 2023-10-20T03:35:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.3136053969999999, 2023-10-20T03:35:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0047625899999999997, 2023-10-20T03:35:36.784 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388819+0000", 2023-10-20T03:35:36.785 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388819+0000", 2023-10-20T03:35:36.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388820+0000", 2023-10-20T03:35:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7e-07 2023-10-20T03:35:36.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388885+0000", 2023-10-20T03:35:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5129999999999995e-05 2023-10-20T03:35:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388926+0000", 2023-10-20T03:35:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0923000000000001e-05 2023-10-20T03:35:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388953+0000", 2023-10-20T03:35:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6842000000000002e-05 2023-10-20T03:35:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.790 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389181+0000", 2023-10-20T03:35:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000228266 2023-10-20T03:35:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389210+0000", 2023-10-20T03:35:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9428999999999999e-05 2023-10-20T03:35:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.393487+0000", 2023-10-20T03:35:36.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0042767350000000003 2023-10-20T03:35:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.393582+0000", 2023-10-20T03:35:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4795000000000003e-05 2023-10-20T03:35:36.793 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.794 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s5 49/31 ECSubWrite(tid=24, reqid=client.4570.0:392, at_version=49'14, trim_to=0'0, roll_forward_to=44'12))", 2023-10-20T03:35:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.389054+0000", 2023-10-20T03:35:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.313370645, 2023-10-20T03:35:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010623927, 2023-10-20T03:35:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.795 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389054+0000", 2023-10-20T03:35:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389054+0000", 2023-10-20T03:35:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.797 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389054+0000", 2023-10-20T03:35:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5999999999999999e-07 2023-10-20T03:35:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389099+0000", 2023-10-20T03:35:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4496000000000002e-05 2023-10-20T03:35:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389126+0000", 2023-10-20T03:35:36.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7515999999999999e-05 2023-10-20T03:35:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389147+0000", 2023-10-20T03:35:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0353999999999999e-05 2023-10-20T03:35:36.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389659+0000", 2023-10-20T03:35:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051199199999999996 2023-10-20T03:35:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.802 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.802 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389687+0000", 2023-10-20T03:35:36.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8374e-05 2023-10-20T03:35:36.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.802 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.399589+0000", 2023-10-20T03:35:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0099021990000000004 2023-10-20T03:35:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.804 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.399678+0000", 2023-10-20T03:35:36.804 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8535999999999994e-05 2023-10-20T03:35:36.804 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.804 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.804 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s5 49/31 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=49'16, trim_to=0'0, roll_forward_to=44'12))", 2023-10-20T03:35:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.393115+0000", 2023-10-20T03:35:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.3093100089999998, 2023-10-20T03:35:36.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011037727000000001, 2023-10-20T03:35:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.806 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.393115+0000", 2023-10-20T03:35:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.393115+0000", 2023-10-20T03:35:36.807 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.393117+0000", 2023-10-20T03:35:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.396e-06 2023-10-20T03:35:36.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.393169+0000", 2023-10-20T03:35:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1975999999999998e-05 2023-10-20T03:35:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.810 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.810 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.393212+0000", 2023-10-20T03:35:36.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3340000000000002e-05 2023-10-20T03:35:36.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.810 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.393257+0000", 2023-10-20T03:35:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5160000000000001e-05 2023-10-20T03:35:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.393427+0000", 2023-10-20T03:35:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000169648 2023-10-20T03:35:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.393454+0000", 2023-10-20T03:35:36.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.711e-05 2023-10-20T03:35:36.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.404043+0000", 2023-10-20T03:35:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010589088 2023-10-20T03:35:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.404152+0000", 2023-10-20T03:35:36.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109009 2023-10-20T03:35:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.fs6 49/47 ECSubWrite(tid=38, reqid=osd.3.0:36, at_version=0'0, trim_to=0'0, roll_forward_to=49'23))", 2023-10-20T03:35:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.398226+0000", 2023-10-20T03:35:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.3041987209999997, 2023-10-20T03:35:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0086867509999999995, 2023-10-20T03:35:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.816 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.817 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.817 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.817 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.398226+0000", 2023-10-20T03:35:36.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.398226+0000", 2023-10-20T03:35:36.818 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.398228+0000", 2023-10-20T03:35:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9199999999999998e-06 2023-10-20T03:35:36.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.398255+0000", 2023-10-20T03:35:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7263e-05 2023-10-20T03:35:36.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.398298+0000", 2023-10-20T03:35:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2562999999999999e-05 2023-10-20T03:35:36.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.398339+0000", 2023-10-20T03:35:36.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1743000000000003e-05 2023-10-20T03:35:36.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.398499+0000", 2023-10-20T03:35:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015995899999999999 2023-10-20T03:35:36.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.398523+0000", 2023-10-20T03:35:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.393e-05 2023-10-20T03:35:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.406817+0000", 2023-10-20T03:35:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0082932739999999998 2023-10-20T03:35:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.406913+0000", 2023-10-20T03:35:36.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6099000000000003e-05 2023-10-20T03:35:36.826 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.826 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.3s2 49/31 ECSubWrite(tid=24, reqid=osd.10.0:24, at_version=49'15, trim_to=0'0, roll_forward_to=47'13))", 2023-10-20T03:35:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.399972+0000", 2023-10-20T03:35:36.827 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.3024523500000003, 2023-10-20T03:35:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0070551219999999996, 2023-10-20T03:35:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.828 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.399972+0000", 2023-10-20T03:35:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.829 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.399972+0000", 2023-10-20T03:35:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.830 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.399973+0000", 2023-10-20T03:35:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.16e-07 2023-10-20T03:35:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.831 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.400035+0000", 2023-10-20T03:35:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1703000000000002e-05 2023-10-20T03:35:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.832 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.400067+0000", 2023-10-20T03:35:36.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2212999999999998e-05 2023-10-20T03:35:36.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.400099+0000", 2023-10-20T03:35:36.834 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2487000000000003e-05 2023-10-20T03:35:36.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.834 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.401291+0000", 2023-10-20T03:35:36.835 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0011919439999999999 2023-10-20T03:35:36.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.835 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.835 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.401319+0000", 2023-10-20T03:35:36.836 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7483000000000001e-05 2023-10-20T03:35:36.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.836 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.836 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.406908+0000", 2023-10-20T03:35:36.836 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055890870000000004 2023-10-20T03:35:36.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.837 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.407027+0000", 2023-10-20T03:35:36.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119389 2023-10-20T03:35:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/47 ECSubWrite(tid=118, reqid=osd.13.0:118, at_version=49'29, trim_to=0'0, roll_forward_to=48'27))", 2023-10-20T03:35:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.405505+0000", 2023-10-20T03:35:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.2969197689999996, 2023-10-20T03:35:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090491649999999996, 2023-10-20T03:35:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.839 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.840 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.840 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.840 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.840 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405505+0000", 2023-10-20T03:35:36.840 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.841 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.841 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405505+0000", 2023-10-20T03:35:36.841 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405506+0000", 2023-10-20T03:35:36.842 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1999999999999999e-06 2023-10-20T03:35:36.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405556+0000", 2023-10-20T03:35:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9759000000000003e-05 2023-10-20T03:35:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405590+0000", 2023-10-20T03:35:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3917000000000001e-05 2023-10-20T03:35:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405613+0000", 2023-10-20T03:35:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3303e-05 2023-10-20T03:35:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.846 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405829+0000", 2023-10-20T03:35:36.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000215952 2023-10-20T03:35:36.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.846 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405856+0000", 2023-10-20T03:35:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6835999999999999e-05 2023-10-20T03:35:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.414449+0000", 2023-10-20T03:35:36.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0085928879999999999 2023-10-20T03:35:36.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.848 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.848 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.414554+0000", 2023-10-20T03:35:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010531000000000001 2023-10-20T03:35:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.850 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s5 49/31 ECSubWrite(tid=26, reqid=osd.8.0:25, at_version=0'0, trim_to=0'0, roll_forward_to=49'16))", 2023-10-20T03:35:36.850 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.405783+0000", 2023-10-20T03:35:36.850 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.2966419079999998, 2023-10-20T03:35:36.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0059182849999999997, 2023-10-20T03:35:36.850 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405783+0000", 2023-10-20T03:35:36.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405783+0000", 2023-10-20T03:35:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405784+0000", 2023-10-20T03:35:36.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4339999999999999e-06 2023-10-20T03:35:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405810+0000", 2023-10-20T03:35:36.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6173000000000002e-05 2023-10-20T03:35:36.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.855 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.855 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405836+0000", 2023-10-20T03:35:36.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6109e-05 2023-10-20T03:35:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.405859+0000", 2023-10-20T03:35:36.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2427000000000001e-05 2023-10-20T03:35:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.406067+0000", 2023-10-20T03:35:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020831499999999999 2023-10-20T03:35:36.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.406091+0000", 2023-10-20T03:35:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3462999999999999e-05 2023-10-20T03:35:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.411598+0000", 2023-10-20T03:35:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055076719999999999 2023-10-20T03:35:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.411701+0000", 2023-10-20T03:35:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000102692 2023-10-20T03:35:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.860 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.3s2 49/31 ECSubWrite(tid=25, reqid=osd.10.0:24, at_version=0'0, trim_to=0'0, roll_forward_to=49'15))", 2023-10-20T03:35:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.407825+0000", 2023-10-20T03:35:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.2945991970000001, 2023-10-20T03:35:36.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0072123389999999999, 2023-10-20T03:35:36.862 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.862 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.862 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.862 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.407825+0000", 2023-10-20T03:35:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.863 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.407825+0000", 2023-10-20T03:35:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.407827+0000", 2023-10-20T03:35:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.813e-06 2023-10-20T03:35:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.407857+0000", 2023-10-20T03:35:36.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.938e-05 2023-10-20T03:35:36.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.407886+0000", 2023-10-20T03:35:36.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9836999999999999e-05 2023-10-20T03:35:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.407908+0000", 2023-10-20T03:35:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1126999999999999e-05 2023-10-20T03:35:36.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.408060+0000", 2023-10-20T03:35:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000152275 2023-10-20T03:35:36.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.408085+0000", 2023-10-20T03:35:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5089e-05 2023-10-20T03:35:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.414939+0000", 2023-10-20T03:35:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0068541619999999996 2023-10-20T03:35:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.415038+0000", 2023-10-20T03:35:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8655999999999999e-05 2023-10-20T03:35:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.871 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.872 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.cs5 49/47 ECSubWrite(tid=122, reqid=osd.13.0:120, at_version=0'0, trim_to=0'0, roll_forward_to=49'31))", 2023-10-20T03:35:36.872 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.420247+0000", 2023-10-20T03:35:36.872 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.2821771269999998, 2023-10-20T03:35:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0081438450000000003, 2023-10-20T03:35:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.420247+0000", 2023-10-20T03:35:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.874 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.420247+0000", 2023-10-20T03:35:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.875 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.420250+0000", 2023-10-20T03:35:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.074e-06 2023-10-20T03:35:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.876 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.420307+0000", 2023-10-20T03:35:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7586000000000001e-05 2023-10-20T03:35:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.420355+0000", 2023-10-20T03:35:36.877 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8136000000000001e-05 2023-10-20T03:35:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.420381+0000", 2023-10-20T03:35:36.878 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5463e-05 2023-10-20T03:35:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.420635+0000", 2023-10-20T03:35:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025434200000000001 2023-10-20T03:35:36.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.420662+0000", 2023-10-20T03:35:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6990000000000001e-05 2023-10-20T03:35:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.428319+0000", 2023-10-20T03:35:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0076564950000000001 2023-10-20T03:35:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.881 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.428391+0000", 2023-10-20T03:35:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2758999999999999e-05 2023-10-20T03:35:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.882 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s5 52/31 ECSubWrite(tid=31, reqid=osd.8.0:31, at_version=52'22, trim_to=0'0, roll_forward_to=51'20))", 2023-10-20T03:35:36.883 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:35.285247+0000", 2023-10-20T03:35:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.4171770960000001, 2023-10-20T03:35:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050199520000000003, 2023-10-20T03:35:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.285247+0000", 2023-10-20T03:35:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.885 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.285247+0000", 2023-10-20T03:35:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.886 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.285248+0000", 2023-10-20T03:35:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6399999999999998e-07 2023-10-20T03:35:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.887 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.285299+0000", 2023-10-20T03:35:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1433e-05 2023-10-20T03:35:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.888 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.285339+0000", 2023-10-20T03:35:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9428999999999998e-05 2023-10-20T03:35:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.285366+0000", 2023-10-20T03:35:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.688e-05 2023-10-20T03:35:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.285534+0000", 2023-10-20T03:35:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016886200000000001 2023-10-20T03:35:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.285565+0000", 2023-10-20T03:35:36.891 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0443e-05 2023-10-20T03:35:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.290165+0000", 2023-10-20T03:35:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0046005860000000003 2023-10-20T03:35:36.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.290267+0000", 2023-10-20T03:35:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000101955 2023-10-20T03:35:36.893 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.894 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.894 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.894 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.894 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s5 53/31 ECSubWrite(tid=33, reqid=osd.8.0:33, at_version=53'24, trim_to=0'0, roll_forward_to=52'22))", 2023-10-20T03:35:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:35.430557+0000", 2023-10-20T03:35:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.271867689, 2023-10-20T03:35:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053525930000000001, 2023-10-20T03:35:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.895 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.430557+0000", 2023-10-20T03:35:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.430557+0000", 2023-10-20T03:35:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.897 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.898 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.898 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.430558+0000", 2023-10-20T03:35:36.898 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5400000000000001e-06 2023-10-20T03:35:36.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.430613+0000", 2023-10-20T03:35:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4685999999999998e-05 2023-10-20T03:35:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.899 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.430641+0000", 2023-10-20T03:35:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7574000000000001e-05 2023-10-20T03:35:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.900 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.430666+0000", 2023-10-20T03:35:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5395999999999999e-05 2023-10-20T03:35:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.901 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.430904+0000", 2023-10-20T03:35:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00023797799999999999 2023-10-20T03:35:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.430988+0000", 2023-10-20T03:35:36.902 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4393000000000003e-05 2023-10-20T03:35:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.435813+0000", 2023-10-20T03:35:36.903 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0048250330000000003 2023-10-20T03:35:36.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.904 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.904 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.435909+0000", 2023-10-20T03:35:36.904 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5993000000000001e-05 2023-10-20T03:35:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.905 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.fs6 53/47 ECSubWrite(tid=57, reqid=client.4570.0:408, at_version=53'31, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:36.906 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:35.448443+0000", 2023-10-20T03:35:36.906 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.253981869, 2023-10-20T03:35:36.906 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015353739, 2023-10-20T03:35:36.906 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.906 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.448443+0000", 2023-10-20T03:35:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.909 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.909 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.448443+0000", 2023-10-20T03:35:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.448445+0000", 2023-10-20T03:35:36.910 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3999999999999999e-06 2023-10-20T03:35:36.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.911 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.911 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.449247+0000", 2023-10-20T03:35:36.911 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00080198499999999996 2023-10-20T03:35:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.449274+0000", 2023-10-20T03:35:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7336999999999999e-05 2023-10-20T03:35:36.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.449289+0000", 2023-10-20T03:35:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4222999999999999e-05 2023-10-20T03:35:36.913 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.914 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.914 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.449396+0000", 2023-10-20T03:35:36.914 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107756 2023-10-20T03:35:36.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.449416+0000", 2023-10-20T03:35:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9606000000000002e-05 2023-10-20T03:35:36.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.463712+0000", 2023-10-20T03:35:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014295809 2023-10-20T03:35:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.463796+0000", 2023-10-20T03:35:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4623000000000006e-05 2023-10-20T03:35:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.917 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.fs6 53/47 ECSubWrite(tid=58, reqid=client.4570.0:409, at_version=53'32, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:35.472532+0000", 2023-10-20T03:35:36.919 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.229892569, 2023-10-20T03:35:36.919 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016132318, 2023-10-20T03:35:36.919 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:36.919 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:36.919 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.472532+0000", 2023-10-20T03:35:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.472532+0000", 2023-10-20T03:35:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.922 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:36.922 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.472534+0000", 2023-10-20T03:35:36.922 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.643e-06 2023-10-20T03:35:36.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.472866+0000", 2023-10-20T03:35:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00033227400000000002 2023-10-20T03:35:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.472897+0000", 2023-10-20T03:35:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1106999999999999e-05 2023-10-20T03:35:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.924 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.472912+0000", 2023-10-20T03:35:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5322999999999999e-05 2023-10-20T03:35:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.925 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:36.926 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.473029+0000", 2023-10-20T03:35:36.926 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011634299999999999 2023-10-20T03:35:36.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.473046+0000", 2023-10-20T03:35:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7546000000000001e-05 2023-10-20T03:35:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.927 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.488610+0000", 2023-10-20T03:35:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015563926 2023-10-20T03:35:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:36.928 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.488664+0000", 2023-10-20T03:35:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4156000000000001e-05 2023-10-20T03:35:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.929 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:36.930 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:36.930 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:36.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-10-20T03:35:36.932 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:36.928+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:37.032 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:37.028+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:37.062 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:35:37.109 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:37.109 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:37.109 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:35:37.109 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:37.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-10-20T03:35:37.133 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:37.132+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:37.223 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:37.223 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.234 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:37.232+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:37.279 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:37.279 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.295 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:37.296 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:37.296 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:35:37.296 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:35:37.296 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:37.307 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:37.307 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.310 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-10-20T03:35:37.335 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:37.332+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:37.337 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:37.337 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:37.363 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.389 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:37.389 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.390 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:37.415 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:37.415 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.434 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:37.434 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.436 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:37.436+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:37.436 INFO:tasks.rados.rados.0.smithi170.stdout:174: done (5 left) 2023-10-20T03:35:37.436 INFO:tasks.rados.rados.0.smithi170.stdout:176: done (4 left) 2023-10-20T03:35:37.437 INFO:tasks.rados.rados.0.smithi170.stdout:178: done (3 left) 2023-10-20T03:35:37.437 INFO:tasks.rados.rados.0.smithi170.stdout:179: rollback oid 31 current snap is 12 2023-10-20T03:35:37.437 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 31 to 11 2023-10-20T03:35:37.437 INFO:tasks.rados.rados.0.smithi170.stdout:180: setattr oid 40 current snap is 12 2023-10-20T03:35:37.438 INFO:tasks.rados.rados.0.smithi170.stdout:181: rmattr oid 46 current snap is 12 2023-10-20T03:35:37.438 INFO:tasks.rados.rados.0.smithi170.stdout:181: done (5 left) 2023-10-20T03:35:37.438 INFO:tasks.rados.rados.0.smithi170.stdout:182: setattr oid 42 current snap is 12 2023-10-20T03:35:37.440 INFO:tasks.rados.rados.0.smithi170.stdout:183: snap_remove snap 0 2023-10-20T03:35:37.446 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:37.446 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:37.470 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-10-20T03:35:37.479 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:37.479 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.505 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:37.506 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.514 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-10-20T03:35:37.532 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:37.532 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.536 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:37.532+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:37.551 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:37.552 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.574 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:37.574 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.607 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:37.607 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:37.610 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:37.610 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.621 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-10-20T03:35:37.637 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:37.636+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:37.646 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:37.647 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:35:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:37.652 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:37.738 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:37.736+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:37.787 INFO:tasks.rados.rados.0.smithi170.stdout:183: done (6 left) 2023-10-20T03:35:37.787 INFO:tasks.rados.rados.0.smithi170.stdout:184: rollback oid 49 current snap is 12 2023-10-20T03:35:37.788 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 49 to 11 2023-10-20T03:35:37.788 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 42 v 27 (ObjNum 60 snap 2 seq_num 60) dirty exists 2023-10-20T03:35:37.788 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 22 (ObjNum 39 snap 0 seq_num 39) dirty exists 2023-10-20T03:35:37.788 INFO:tasks.rados.rados.0.smithi170.stdout:179: finishing rollback tid 0 to smithi17039300-31 2023-10-20T03:35:37.789 INFO:tasks.rados.rados.0.smithi170.stdout:179: finishing rollback tid 1 to smithi17039300-31 2023-10-20T03:35:37.789 INFO:tasks.rados.rados.0.smithi170.stdout:179: finishing rollback tid 2 to smithi17039300-31 2023-10-20T03:35:37.789 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 31 v 40 (ObjNum 78 snap 7 seq_num 78) dirty exists 2023-10-20T03:35:37.789 INFO:tasks.rados.rados.0.smithi170.stdout:175: finishing rollback tid 0 to smithi17039300-35 2023-10-20T03:35:37.789 INFO:tasks.rados.rados.0.smithi170.stdout:175: finishing rollback tid 1 to smithi17039300-35 2023-10-20T03:35:37.790 INFO:tasks.rados.rados.0.smithi170.stdout:175: finishing rollback tid 2 to smithi17039300-35 2023-10-20T03:35:37.790 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 42 (ObjNum 34 snap 0 seq_num 34) dirty exists 2023-10-20T03:35:37.790 INFO:tasks.rados.rados.0.smithi170.stdout:171: finishing copy_from racing read to smithi17039300-8 2023-10-20T03:35:37.790 INFO:tasks.rados.rados.0.smithi170.stdout:171: finishing copy_from to smithi17039300-8 2023-10-20T03:35:37.791 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 8 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-10-20T03:35:37.791 INFO:tasks.rados.rados.0.smithi170.stdout:177: finishing write tid 1 to smithi17039300-50 2023-10-20T03:35:37.791 INFO:tasks.rados.rados.0.smithi170.stdout:177: finishing write tid 2 to smithi17039300-50 2023-10-20T03:35:37.791 INFO:tasks.rados.rados.0.smithi170.stdout:177: finishing write tid 3 to smithi17039300-50 2023-10-20T03:35:37.791 INFO:tasks.rados.rados.0.smithi170.stdout:177: finishing write tid 4 to smithi17039300-50 2023-10-20T03:35:37.792 INFO:tasks.rados.rados.0.smithi170.stdout:177: finishing write tid 5 to smithi17039300-50 2023-10-20T03:35:37.792 INFO:tasks.rados.rados.0.smithi170.stdout:177: oid 50 updating version 0 to 23 2023-10-20T03:35:37.792 INFO:tasks.rados.rados.0.smithi170.stdout:177: oid 50 version 23 is already newer than 21 2023-10-20T03:35:37.792 INFO:tasks.rados.rados.0.smithi170.stdout:177: oid 50 version 23 is already newer than 22 2023-10-20T03:35:37.792 INFO:tasks.rados.rados.0.smithi170.stdout:177: oid 50 updating version 23 to 24 2023-10-20T03:35:37.793 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 50 v 24 (ObjNum 92 snap 11 seq_num 92) dirty exists 2023-10-20T03:35:37.793 INFO:tasks.rados.rados.0.smithi170.stdout:177: left oid 50 (ObjNum 92 snap 11 seq_num 92) 2023-10-20T03:35:37.793 INFO:tasks.rados.rados.0.smithi170.stdout:171: done (6 left) 2023-10-20T03:35:37.793 INFO:tasks.rados.rados.0.smithi170.stdout:175: done (5 left) 2023-10-20T03:35:37.793 INFO:tasks.rados.rados.0.smithi170.stdout:177: done (4 left) 2023-10-20T03:35:37.794 INFO:tasks.rados.rados.0.smithi170.stdout:179: done (3 left) 2023-10-20T03:35:37.794 INFO:tasks.rados.rados.0.smithi170.stdout:180: done (2 left) 2023-10-20T03:35:37.794 INFO:tasks.rados.rados.0.smithi170.stdout:182: done (1 left) 2023-10-20T03:35:37.794 INFO:tasks.rados.rados.0.smithi170.stdout:185: delete oid 41 current snap is 12 2023-10-20T03:35:37.796 INFO:tasks.rados.rados.0.smithi170.stdout:184: finishing rollback tid 0 to smithi17039300-49 2023-10-20T03:35:37.797 INFO:tasks.rados.rados.0.smithi170.stdout:184: finishing rollback tid 1 to smithi17039300-49 2023-10-20T03:35:37.799 INFO:tasks.rados.rados.0.smithi170.stdout:184: finishing rollback tid 2 to smithi17039300-49 2023-10-20T03:35:37.799 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 49 v 39 (ObjNum 91 snap 11 seq_num 91) dirty exists 2023-10-20T03:35:37.799 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:37.800 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:37.814 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-10-20T03:35:37.839 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:37.834+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:37.937 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:37.937 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:37.940 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:37.938+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:37.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-10-20T03:35:38.040 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:38.040+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:38.071 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:38.071 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:38.086 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-10-20T03:35:38.141 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:38.140+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:38.210 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:38.211 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:38.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-10-20T03:35:38.242 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:38.238+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:38.343 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:38.341+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:38.353 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:38.353 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:38.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-10-20T03:35:38.444 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:38.440+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:38.501 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:38.501 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:38.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-10-20T03:35:38.545 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:38.542+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:38.639 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:38.639 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:38.645 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:38.641+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:38.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-10-20T03:35:38.746 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:38.742+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:38.778 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:38.779 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2023-10-20T03:35:38.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-10-20T03:35:38.821 INFO:tasks.rados.rados.0.smithi170.stdout:184: done (1 left) 2023-10-20T03:35:38.822 INFO:tasks.rados.rados.0.smithi170.stdout:185: done (0 left) 2023-10-20T03:35:38.822 INFO:tasks.rados.rados.0.smithi170.stdout:186: read oid 40 snap -1 2023-10-20T03:35:38.822 INFO:tasks.rados.rados.0.smithi170.stdout:186: expect (ObjNum 39 snap 0 seq_num 39) 2023-10-20T03:35:38.822 INFO:tasks.rados.rados.0.smithi170.stdout:187: read oid 3 snap -1 2023-10-20T03:35:38.823 INFO:tasks.rados.rados.0.smithi170.stdout:187: expect (ObjNum 2 snap 0 seq_num 2) 2023-10-20T03:35:38.823 INFO:tasks.rados.rados.0.smithi170.stdout:188: rollback oid 19 current snap is 12 2023-10-20T03:35:38.823 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 19 to 1 2023-10-20T03:35:38.823 INFO:tasks.rados.rados.0.smithi170.stdout:189: snap_remove snap 11 2023-10-20T03:35:38.847 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:38.842+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:38.926 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:38.926 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:38.940 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-10-20T03:35:38.984 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:38.984+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:39.049 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:39.044+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:39.067 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:39.067 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:39.082 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-10-20T03:35:39.149 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:39.144+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:39.251 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:39.251 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:39.251 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:39.248+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:39.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-10-20T03:35:39.351 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:39.348+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:39.393 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:39.393 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:39.407 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-10-20T03:35:39.453 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:39.448+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:39.554 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:39.554+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:39.567 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:39.567 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:39.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-10-20T03:35:39.653 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:35:39.655 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:39.652+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:39.700 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:39.700 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:39.714 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-10-20T03:35:39.759 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:39.758+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:39.799 INFO:tasks.rados.rados.0.smithi170.stdout:189: done (3 left) 2023-10-20T03:35:39.799 INFO:tasks.rados.rados.0.smithi170.stdout:190: snap_create 2023-10-20T03:35:39.800 INFO:tasks.rados.rados.0.smithi170.stdout:188: finishing rollback tid 1 to smithi17039300-19 2023-10-20T03:35:39.800 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 0 (ObjNum 129 snap 0 seq_num 3159993848) dirty dne 2023-10-20T03:35:39.800 INFO:tasks.rados.rados.0.smithi170.stdout:186: expect (ObjNum 39 snap 0 seq_num 39) 2023-10-20T03:35:39.812 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:39.812 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:35:39.832 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:39.833 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:35:39.847 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:39.847 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:39.852 INFO:tasks.rados.rados.0.smithi170.stdout:187: expect (ObjNum 2 snap 0 seq_num 2) 2023-10-20T03:35:39.854 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:39.855 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:35:39.857 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:39.853+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:39.861 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-10-20T03:35:39.875 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:39.875 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:35:39.893 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:39.893 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:35:39.915 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:39.916 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:35:39.916 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:39.939 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:39.939 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:35:39.958 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:39.954+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:39.958 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:39.958 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:35:39.981 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:39.982 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-10-20T03:35:40.010 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:40.011 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:35:40.026 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:40.027 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:40.027 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:35:40.027 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:35:40.027 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:40.032 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:40.033 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:35:40.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-10-20T03:35:40.052 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:40.053 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:35:40.058 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:40.056+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:40.075 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:40.075 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:35:40.095 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:40.096 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:35:40.115 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:40.115 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:35:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:40.122 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:40.159 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:40.156+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:40.207 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:40.208 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2023-10-20T03:35:40.208 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2023-10-20T03:35:40.208 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2023-10-20T03:35:40.208 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.209 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.2s1 36/31 ECSubWrite(tid=21, reqid=client.4570.0:198, at_version=36'13, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:40.209 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:18.265621+0000", 2023-10-20T03:35:40.209 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.940989542000001, 2023-10-20T03:35:40.209 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027667458999999998, 2023-10-20T03:35:40.210 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.210 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:40.210 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.210 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.210 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.210 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.265621+0000", 2023-10-20T03:35:40.211 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.211 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.211 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.211 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.211 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.265621+0000", 2023-10-20T03:35:40.212 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.212 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.212 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.212 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.212 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.265622+0000", 2023-10-20T03:35:40.213 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.06e-07 2023-10-20T03:35:40.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.213 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.213 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.213 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.266264+0000", 2023-10-20T03:35:40.213 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00064158200000000002 2023-10-20T03:35:40.214 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.214 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.214 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.214 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.266298+0000", 2023-10-20T03:35:40.215 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4239000000000002e-05 2023-10-20T03:35:40.215 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.215 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.215 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.215 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.266330+0000", 2023-10-20T03:35:40.216 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2061999999999999e-05 2023-10-20T03:35:40.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.216 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.216 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.216 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.266516+0000", 2023-10-20T03:35:40.217 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018592200000000001 2023-10-20T03:35:40.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.217 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.218 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.266547+0000", 2023-10-20T03:35:40.218 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.1556000000000001e-05 2023-10-20T03:35:40.218 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.218 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.219 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.293239+0000", 2023-10-20T03:35:40.219 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026692156000000002 2023-10-20T03:35:40.219 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.219 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.219 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.220 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.293289+0000", 2023-10-20T03:35:40.220 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9236000000000001e-05 2023-10-20T03:35:40.220 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.220 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.220 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.221 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.221 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.1s5 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:40.221 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:25.329996+0000", 2023-10-20T03:35:40.221 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.876614296, 2023-10-20T03:35:40.222 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025961209999999998, 2023-10-20T03:35:40.222 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.222 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:40.222 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.222 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.223 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.223 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.329996+0000", 2023-10-20T03:35:40.223 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.223 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.223 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.224 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.224 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.329996+0000", 2023-10-20T03:35:40.224 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.225 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.225 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.329997+0000", 2023-10-20T03:35:40.225 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5700000000000001e-07 2023-10-20T03:35:40.225 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.225 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.226 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.226 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330568+0000", 2023-10-20T03:35:40.226 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00057021199999999995 2023-10-20T03:35:40.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.227 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.227 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.227 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330605+0000", 2023-10-20T03:35:40.227 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7154999999999999e-05 2023-10-20T03:35:40.228 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.228 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.228 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.228 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330630+0000", 2023-10-20T03:35:40.228 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5435999999999998e-05 2023-10-20T03:35:40.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.229 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.229 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.229 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330868+0000", 2023-10-20T03:35:40.229 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000237956 2023-10-20T03:35:40.230 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.230 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.230 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.230 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330899+0000", 2023-10-20T03:35:40.231 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0608999999999998e-05 2023-10-20T03:35:40.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.231 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.231 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.232 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.355827+0000", 2023-10-20T03:35:40.232 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.024927936000000001 2023-10-20T03:35:40.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.232 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.232 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.233 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.355958+0000", 2023-10-20T03:35:40.233 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000131049 2023-10-20T03:35:40.233 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.233 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.234 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.234 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.234 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.234 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:323 3.0s0 3:0e4cfe35:::smithi17039300-49:head [append 0~589824] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:40.234 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.357685+0000", 2023-10-20T03:35:40.235 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 13.848925425999999, 2023-10-20T03:35:40.235 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039449378, 2023-10-20T03:35:40.235 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.235 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:40.235 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.236 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.236 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.236 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 323 2023-10-20T03:35:40.236 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.236 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.237 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.237 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.237 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.357685+0000", 2023-10-20T03:35:40.237 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.238 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.238 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.238 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.357685+0000", 2023-10-20T03:35:40.238 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.239 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.239 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.239 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.239 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.357688+0000", 2023-10-20T03:35:40.239 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7159999999999999e-06 2023-10-20T03:35:40.240 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.240 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.240 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.240 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358334+0000", 2023-10-20T03:35:40.240 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00064636299999999995 2023-10-20T03:35:40.241 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.241 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.241 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.241 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358368+0000", 2023-10-20T03:35:40.241 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3182000000000002e-05 2023-10-20T03:35:40.242 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.242 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.242 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.242 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358382+0000", 2023-10-20T03:35:40.243 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4797000000000001e-05 2023-10-20T03:35:40.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.243 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.243 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.243 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358559+0000", 2023-10-20T03:35:40.244 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000177057 2023-10-20T03:35:40.244 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.244 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.244 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.244 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358736+0000", 2023-10-20T03:35:40.245 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000176861 2023-10-20T03:35:40.245 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.245 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.245 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.245 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361012+0000", 2023-10-20T03:35:40.246 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002275792 2023-10-20T03:35:40.246 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.246 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.246 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.246 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.396529+0000", 2023-10-20T03:35:40.247 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.035517254999999998 2023-10-20T03:35:40.247 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.247 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.247 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:40.248 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.396672+0000", 2023-10-20T03:35:40.248 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014286600000000001 2023-10-20T03:35:40.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.248 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.248 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.249 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.397135+0000", 2023-10-20T03:35:40.249 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00046248900000000001 2023-10-20T03:35:40.249 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.249 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.249 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.250 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:324 3.0s0 3:0e4cfe35:::smithi17039300-49:head [append 0~573440] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:40.250 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.365103+0000", 2023-10-20T03:35:40.250 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 13.841507642, 2023-10-20T03:35:40.251 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.045538596000000001, 2023-10-20T03:35:40.251 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.251 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:40.251 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.251 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.252 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.252 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 324 2023-10-20T03:35:40.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.252 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.253 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.253 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365103+0000", 2023-10-20T03:35:40.253 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.253 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.254 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.254 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365103+0000", 2023-10-20T03:35:40.254 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.255 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.255 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365105+0000", 2023-10-20T03:35:40.255 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3e-06 2023-10-20T03:35:40.255 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.256 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365744+0000", 2023-10-20T03:35:40.256 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000638213 2023-10-20T03:35:40.256 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.257 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.257 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365777+0000", 2023-10-20T03:35:40.257 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3522000000000002e-05 2023-10-20T03:35:40.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.258 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.258 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.258 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365792+0000", 2023-10-20T03:35:40.258 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5227e-05 2023-10-20T03:35:40.258 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.259 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.259 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.259 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.379267+0000", 2023-10-20T03:35:40.259 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013474832000000001 2023-10-20T03:35:40.259 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.260 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.260 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.379462+0000", 2023-10-20T03:35:40.260 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019448100000000001 2023-10-20T03:35:40.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.261 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.261 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.261 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.381551+0000", 2023-10-20T03:35:40.261 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002089034 2023-10-20T03:35:40.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.262 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.262 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.404299+0000", 2023-10-20T03:35:40.262 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.022748244000000001 2023-10-20T03:35:40.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:40.263 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.410190+0000", 2023-10-20T03:35:40.263 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0058907630000000002 2023-10-20T03:35:40.264 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.264 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.264 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.264 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.410642+0000", 2023-10-20T03:35:40.265 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00045197999999999999 2023-10-20T03:35:40.265 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.265 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.265 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.266 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.266 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:325 3.0s0 3:0e4cfe35:::smithi17039300-49:head [append 0~196608] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:40.266 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.367541+0000", 2023-10-20T03:35:40.266 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 13.839069489, 2023-10-20T03:35:40.266 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.044806372999999997, 2023-10-20T03:35:40.267 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.267 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:40.267 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.267 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.268 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.268 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 325 2023-10-20T03:35:40.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.268 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.268 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.269 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.269 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367541+0000", 2023-10-20T03:35:40.269 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.269 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.270 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.270 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367541+0000", 2023-10-20T03:35:40.270 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.270 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.271 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.271 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.271 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367544+0000", 2023-10-20T03:35:40.271 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6129999999999998e-06 2023-10-20T03:35:40.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.272 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.272 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367692+0000", 2023-10-20T03:35:40.272 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014814900000000001 2023-10-20T03:35:40.272 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.273 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.273 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.273 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367725+0000", 2023-10-20T03:35:40.273 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2752000000000001e-05 2023-10-20T03:35:40.273 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.274 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.274 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367759+0000", 2023-10-20T03:35:40.274 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4654999999999999e-05 2023-10-20T03:35:40.275 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.275 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.275 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.275 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.397445+0000", 2023-10-20T03:35:40.275 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029685547 2023-10-20T03:35:40.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.276 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.276 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.397646+0000", 2023-10-20T03:35:40.276 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020082300000000001 2023-10-20T03:35:40.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.277 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.277 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:35:40.277 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001263 2023-10-20T03:35:40.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.278 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.278 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.405032+0000", 2023-10-20T03:35:40.278 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0061230240000000003 2023-10-20T03:35:40.279 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.279 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.279 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:40.279 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.412065+0000", 2023-10-20T03:35:40.280 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.007033437 2023-10-20T03:35:40.280 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.280 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.280 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.280 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.412348+0000", 2023-10-20T03:35:40.280 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00028237299999999999 2023-10-20T03:35:40.281 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.281 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.281 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.282 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:326 3.0s0 3:0e4cfe35:::smithi17039300-49:head [setxattr _header (56) in=7b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:40.282 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.367791+0000", 2023-10-20T03:35:40.282 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 13.838820131, 2023-10-20T03:35:40.282 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.045708658999999999, 2023-10-20T03:35:40.282 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.283 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:40.283 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.283 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.283 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.283 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 326 2023-10-20T03:35:40.284 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.284 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.284 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.284 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.284 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367791+0000", 2023-10-20T03:35:40.284 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.285 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.285 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.285 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.285 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367791+0000", 2023-10-20T03:35:40.286 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.286 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.286 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367792+0000", 2023-10-20T03:35:40.287 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1370000000000001e-06 2023-10-20T03:35:40.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.287 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.287 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367798+0000", 2023-10-20T03:35:40.288 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1229999999999997e-06 2023-10-20T03:35:40.288 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.288 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.288 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.289 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367813+0000", 2023-10-20T03:35:40.289 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5130000000000001e-05 2023-10-20T03:35:40.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.289 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.290 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367823+0000", 2023-10-20T03:35:40.290 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0103e-05 2023-10-20T03:35:40.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.290 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.291 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402753+0000", 2023-10-20T03:35:40.291 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.034930226000000002 2023-10-20T03:35:40.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402936+0000", 2023-10-20T03:35:40.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000182542 2023-10-20T03:35:40.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.403933+0000", 2023-10-20T03:35:40.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00099678799999999997 2023-10-20T03:35:40.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.294 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.294 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.405527+0000", 2023-10-20T03:35:40.294 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001594332 2023-10-20T03:35:40.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.295 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:40.295 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.413173+0000", 2023-10-20T03:35:40.295 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0076458400000000001 2023-10-20T03:35:40.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.296 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.296 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.296 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.413499+0000", 2023-10-20T03:35:40.296 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00032643800000000002 2023-10-20T03:35:40.296 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.297 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.297 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.297 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:327 3.0s0 3:0e4cfe35:::smithi17039300-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:40.298 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.367835+0000", 2023-10-20T03:35:40.298 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 13.838775375000001, 2023-10-20T03:35:40.298 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.053408074999999999, 2023-10-20T03:35:40.298 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.298 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:40.299 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.299 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.299 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.299 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 327 2023-10-20T03:35:40.299 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.300 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.300 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.300 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.300 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367835+0000", 2023-10-20T03:35:40.300 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.300 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.301 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.301 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.301 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367835+0000", 2023-10-20T03:35:40.301 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.301 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.302 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.302 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.302 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367836+0000", 2023-10-20T03:35:40.302 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3699999999999999e-07 2023-10-20T03:35:40.302 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.303 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.303 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.303 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367840+0000", 2023-10-20T03:35:40.303 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6200000000000001e-06 2023-10-20T03:35:40.303 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.304 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.304 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367851+0000", 2023-10-20T03:35:40.304 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0764999999999999e-05 2023-10-20T03:35:40.305 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.305 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.305 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.305 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367858+0000", 2023-10-20T03:35:40.305 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7700000000000001e-06 2023-10-20T03:35:40.305 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.306 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.306 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.306 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.404384+0000", 2023-10-20T03:35:40.306 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.036525558999999999 2023-10-20T03:35:40.307 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.307 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.307 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:40.307 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.404538+0000", 2023-10-20T03:35:40.307 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000154 2023-10-20T03:35:40.308 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.308 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.308 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.308 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.414622+0000", 2023-10-20T03:35:40.308 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.01008392 2023-10-20T03:35:40.309 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.309 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.309 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.309 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.414717+0000", 2023-10-20T03:35:40.309 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5476999999999994e-05 2023-10-20T03:35:40.310 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.310 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.310 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.310 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.421243+0000", 2023-10-20T03:35:40.311 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0065260270000000002 2023-10-20T03:35:40.311 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.311 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.311 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.311 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.312 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.312 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 44/31 ECSubWriteReply(tid=18, last_complete=44'9, committed=1, applied=1))", 2023-10-20T03:35:40.312 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.368368+0000", 2023-10-20T03:35:40.312 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 13.838242485, 2023-10-20T03:35:40.312 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027470569, 2023-10-20T03:35:40.312 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.313 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:40.313 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.313 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.313 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.313 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.368368+0000", 2023-10-20T03:35:40.314 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.314 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.314 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.314 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.315 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.368368+0000", 2023-10-20T03:35:40.315 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.315 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.315 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.315 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.316 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.368369+0000", 2023-10-20T03:35:40.316 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.136e-06 2023-10-20T03:35:40.316 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.316 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.316 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.317 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.368382+0000", 2023-10-20T03:35:40.317 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2765999999999999e-05 2023-10-20T03:35:40.317 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.317 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.317 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.317 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.368401+0000", 2023-10-20T03:35:40.318 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8997000000000001e-05 2023-10-20T03:35:40.318 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.318 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.318 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.318 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.368412+0000", 2023-10-20T03:35:40.319 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0526000000000001e-05 2023-10-20T03:35:40.319 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.319 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.319 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.319 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395780+0000", 2023-10-20T03:35:40.320 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027367995999999999 2023-10-20T03:35:40.320 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.320 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.320 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.320 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395839+0000", 2023-10-20T03:35:40.321 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9147999999999997e-05 2023-10-20T03:35:40.321 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.321 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.321 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.321 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.322 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.322 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:390 3.0s0 3:0e4cfe35:::smithi17039300-49:head [copy-from ver 15] snapc f=[f,d,b,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:35:40.322 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:31.382452+0000", 2023-10-20T03:35:40.322 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 8.8241590480000003, 2023-10-20T03:35:40.323 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.059495292999999998, 2023-10-20T03:35:40.323 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.323 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:40.323 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.324 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.324 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.324 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 390 2023-10-20T03:35:40.324 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.324 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.325 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.325 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.325 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382452+0000", 2023-10-20T03:35:40.325 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.325 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.326 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.326 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.326 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382452+0000", 2023-10-20T03:35:40.326 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.327 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.327 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.327 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382455+0000", 2023-10-20T03:35:40.327 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4070000000000002e-06 2023-10-20T03:35:40.328 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.328 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.328 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.328 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382474+0000", 2023-10-20T03:35:40.328 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8422000000000001e-05 2023-10-20T03:35:40.329 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.329 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.329 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.329 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382516+0000", 2023-10-20T03:35:40.329 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2376000000000002e-05 2023-10-20T03:35:40.330 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.330 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.330 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.330 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382534+0000", 2023-10-20T03:35:40.330 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8335999999999999e-05 2023-10-20T03:35:40.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.331 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.331 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.331 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382662+0000", 2023-10-20T03:35:40.331 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000128083 2023-10-20T03:35:40.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.332 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.332 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op must wait for map", 2023-10-20T03:35:40.332 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382676+0000", 2023-10-20T03:35:40.332 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3679999999999999e-05 2023-10-20T03:35:40.333 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.333 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.333 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.333 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.387857+0000", 2023-10-20T03:35:40.333 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0051807490000000001 2023-10-20T03:35:40.333 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.334 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.334 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.334 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.388142+0000", 2023-10-20T03:35:40.334 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000284729 2023-10-20T03:35:40.335 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.335 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.335 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.335 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.413968+0000", 2023-10-20T03:35:40.335 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025826947999999999 2023-10-20T03:35:40.336 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.336 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.336 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.336 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.440223+0000", 2023-10-20T03:35:40.336 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026254116000000001 2023-10-20T03:35:40.336 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.337 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.337 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:40.337 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.441418+0000", 2023-10-20T03:35:40.337 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001195113 2023-10-20T03:35:40.337 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.338 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.338 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.338 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.441947+0000", 2023-10-20T03:35:40.338 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00052933400000000001 2023-10-20T03:35:40.339 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.339 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.339 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.339 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.339 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.340 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:391 3.0s0 3:0e4cfe35:::smithi17039300-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:35:40.340 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:31.382572+0000", 2023-10-20T03:35:40.340 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 8.8240388149999998, 2023-10-20T03:35:40.340 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.056308376, 2023-10-20T03:35:40.341 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.341 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:40.341 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.341 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.341 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.342 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 391 2023-10-20T03:35:40.342 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.342 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.342 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.343 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.343 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382572+0000", 2023-10-20T03:35:40.343 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.343 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.343 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.344 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.344 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382572+0000", 2023-10-20T03:35:40.344 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.344 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.344 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.345 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.345 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382573+0000", 2023-10-20T03:35:40.345 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4130000000000001e-06 2023-10-20T03:35:40.345 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.345 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.346 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.346 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382580+0000", 2023-10-20T03:35:40.346 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0670000000000003e-06 2023-10-20T03:35:40.346 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.346 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.347 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.347 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382603+0000", 2023-10-20T03:35:40.347 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2099999999999998e-05 2023-10-20T03:35:40.347 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.348 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.348 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.348 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382612+0000", 2023-10-20T03:35:40.348 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3519999999999996e-06 2023-10-20T03:35:40.348 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.349 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.349 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.349 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382806+0000", 2023-10-20T03:35:40.349 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019382199999999999 2023-10-20T03:35:40.349 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.350 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.350 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:40.350 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382827+0000", 2023-10-20T03:35:40.350 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1552e-05 2023-10-20T03:35:40.350 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.351 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.351 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.351 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.388779+0000", 2023-10-20T03:35:40.351 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0059519960000000002 2023-10-20T03:35:40.351 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.352 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.352 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for blocked object", 2023-10-20T03:35:40.352 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.388904+0000", 2023-10-20T03:35:40.352 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000124893 2023-10-20T03:35:40.352 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.353 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.353 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.353 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.438358+0000", 2023-10-20T03:35:40.353 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.049453814999999998 2023-10-20T03:35:40.353 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.354 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.354 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.354 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.438533+0000", 2023-10-20T03:35:40.354 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017465199999999999 2023-10-20T03:35:40.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.355 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.355 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.355 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.438880+0000", 2023-10-20T03:35:40.355 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000347714 2023-10-20T03:35:40.355 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.356 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.356 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.356 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.356 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.356 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:408 3.fs1 3:f90c0d61:::smithi17039300-26:head [append 0~753664] snapc f=[d,3,2] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:35:40.357 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.443873+0000", 2023-10-20T03:35:40.357 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.7627379569999997, 2023-10-20T03:35:40.357 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.045012336, 2023-10-20T03:35:40.357 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.357 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:40.358 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.358 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.358 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.358 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 408 2023-10-20T03:35:40.359 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.359 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.359 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.359 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.359 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.443873+0000", 2023-10-20T03:35:40.360 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.360 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.360 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.360 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.360 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.443873+0000", 2023-10-20T03:35:40.361 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.361 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.361 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.361 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.361 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.443876+0000", 2023-10-20T03:35:40.362 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8200000000000001e-06 2023-10-20T03:35:40.362 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.362 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.362 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.363 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.444508+0000", 2023-10-20T03:35:40.363 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00063252700000000002 2023-10-20T03:35:40.363 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.363 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.363 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.364 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.444542+0000", 2023-10-20T03:35:40.364 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4292999999999998e-05 2023-10-20T03:35:40.364 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.364 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.364 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.365 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.444564+0000", 2023-10-20T03:35:40.365 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1066000000000001e-05 2023-10-20T03:35:40.365 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.365 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.365 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.366 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.446120+0000", 2023-10-20T03:35:40.366 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0015568089999999999 2023-10-20T03:35:40.366 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.366 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.366 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.367 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.446540+0000", 2023-10-20T03:35:40.367 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00041965900000000002 2023-10-20T03:35:40.367 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.367 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.367 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.368 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.449079+0000", 2023-10-20T03:35:40.368 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0025392610000000001 2023-10-20T03:35:40.368 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.368 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.368 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.369 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.469859+0000", 2023-10-20T03:35:40.369 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.020779766000000002 2023-10-20T03:35:40.369 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.369 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.369 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:40.370 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.488709+0000", 2023-10-20T03:35:40.370 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.018849798000000001 2023-10-20T03:35:40.370 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.370 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.370 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.371 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.488885+0000", 2023-10-20T03:35:40.371 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017633699999999999 2023-10-20T03:35:40.371 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.371 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.371 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.372 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.372 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.372 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:409 3.fs1 3:f90c0d61:::smithi17039300-26:head [append 0~720896] snapc f=[d,3,2] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:35:40.372 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.456955+0000", 2023-10-20T03:35:40.373 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.7496559000000005, 2023-10-20T03:35:40.373 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039289445999999999, 2023-10-20T03:35:40.373 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.373 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:40.373 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.374 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.374 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.374 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 409 2023-10-20T03:35:40.374 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.374 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.375 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.375 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.377 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.456955+0000", 2023-10-20T03:35:40.377 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.377 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.378 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.378 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.378 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.456955+0000", 2023-10-20T03:35:40.378 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.378 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.379 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.379 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.379 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.456958+0000", 2023-10-20T03:35:40.379 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.253e-06 2023-10-20T03:35:40.379 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.380 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.380 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.380 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.457987+0000", 2023-10-20T03:35:40.380 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0010284510000000001 2023-10-20T03:35:40.381 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.381 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.381 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.381 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458025+0000", 2023-10-20T03:35:40.381 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8566000000000003e-05 2023-10-20T03:35:40.382 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.382 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.382 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.382 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458043+0000", 2023-10-20T03:35:40.382 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.783e-05 2023-10-20T03:35:40.383 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.383 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.383 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.383 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.470966+0000", 2023-10-20T03:35:40.383 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012923244 2023-10-20T03:35:40.384 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.384 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.384 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.384 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.471149+0000", 2023-10-20T03:35:40.384 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000182638 2023-10-20T03:35:40.385 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.385 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.385 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.385 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.472881+0000", 2023-10-20T03:35:40.386 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017317610000000001 2023-10-20T03:35:40.386 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.386 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.386 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.386 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.488407+0000", 2023-10-20T03:35:40.387 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015526521 2023-10-20T03:35:40.387 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.387 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.387 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:40.387 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.496153+0000", 2023-10-20T03:35:40.388 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0077454489999999997 2023-10-20T03:35:40.388 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.388 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.388 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.388 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.496244+0000", 2023-10-20T03:35:40.389 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1732999999999997e-05 2023-10-20T03:35:40.389 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.389 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.389 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.390 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.390 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.390 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:410 3.fs1 3:f90c0d61:::smithi17039300-26:head [append 0~49152] snapc f=[d,3,2] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:35:40.390 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.458073+0000", 2023-10-20T03:35:40.390 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.7485377150000003, 2023-10-20T03:35:40.391 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039280043000000001, 2023-10-20T03:35:40.391 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.391 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:40.391 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.392 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.392 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.392 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 410 2023-10-20T03:35:40.392 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.392 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.393 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.393 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.393 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458073+0000", 2023-10-20T03:35:40.393 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.393 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.394 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.394 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.394 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458073+0000", 2023-10-20T03:35:40.394 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.395 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.395 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.395 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.395 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458074+0000", 2023-10-20T03:35:40.395 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2100000000000001e-06 2023-10-20T03:35:40.396 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.396 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.396 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.396 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458151+0000", 2023-10-20T03:35:40.396 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6542e-05 2023-10-20T03:35:40.397 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.397 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.397 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.397 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458168+0000", 2023-10-20T03:35:40.397 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7552999999999999e-05 2023-10-20T03:35:40.398 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.398 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.398 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.398 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458180+0000", 2023-10-20T03:35:40.399 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1826e-05 2023-10-20T03:35:40.399 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.399 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.399 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.399 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.485589+0000", 2023-10-20T03:35:40.400 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027409103000000001 2023-10-20T03:35:40.400 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.400 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.400 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.401 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.485726+0000", 2023-10-20T03:35:40.401 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000136482 2023-10-20T03:35:40.401 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.401 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.401 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.402 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.486676+0000", 2023-10-20T03:35:40.402 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000949766 2023-10-20T03:35:40.402 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.402 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.402 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.403 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.491210+0000", 2023-10-20T03:35:40.403 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0045347230000000001 2023-10-20T03:35:40.403 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.403 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.404 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:40.404 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.497284+0000", 2023-10-20T03:35:40.404 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0060733059999999997 2023-10-20T03:35:40.404 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.404 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.405 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.405 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.497353+0000", 2023-10-20T03:35:40.405 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9531999999999998e-05 2023-10-20T03:35:40.405 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.405 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.406 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.406 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.406 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.406 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:411 3.fs1 3:f90c0d61:::smithi17039300-26:head [setxattr _header (57) in=7b] snapc f=[d,3,2] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:35:40.407 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.458196+0000", 2023-10-20T03:35:40.407 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.7484148140000002, 2023-10-20T03:35:40.407 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039654613999999998, 2023-10-20T03:35:40.407 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.407 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:40.408 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.408 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.408 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.408 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 411 2023-10-20T03:35:40.408 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.409 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.409 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.409 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.409 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458196+0000", 2023-10-20T03:35:40.409 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.410 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.410 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.410 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.410 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458196+0000", 2023-10-20T03:35:40.410 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.411 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.411 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.411 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.411 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458197+0000", 2023-10-20T03:35:40.412 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1299999999999998e-07 2023-10-20T03:35:40.412 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.412 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.412 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.412 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458202+0000", 2023-10-20T03:35:40.413 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2170000000000002e-06 2023-10-20T03:35:40.413 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.413 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.413 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.413 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458212+0000", 2023-10-20T03:35:40.414 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0000000000000001e-05 2023-10-20T03:35:40.414 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.414 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.414 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.414 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458220+0000", 2023-10-20T03:35:40.415 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4560000000000003e-06 2023-10-20T03:35:40.415 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.415 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.415 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.416 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.489002+0000", 2023-10-20T03:35:40.416 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030782868000000001 2023-10-20T03:35:40.416 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.416 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.416 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.417 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.489245+0000", 2023-10-20T03:35:40.417 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024254100000000001 2023-10-20T03:35:40.417 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.417 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.418 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.418 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.490266+0000", 2023-10-20T03:35:40.418 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001020721 2023-10-20T03:35:40.418 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.418 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.419 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.419 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.491971+0000", 2023-10-20T03:35:40.419 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017054940000000001 2023-10-20T03:35:40.419 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.420 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.420 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:40.420 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.497768+0000", 2023-10-20T03:35:40.420 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0057964269999999998 2023-10-20T03:35:40.420 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.421 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.421 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.421 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.497851+0000", 2023-10-20T03:35:40.421 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2977000000000002e-05 2023-10-20T03:35:40.421 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.422 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.422 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.422 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.422 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.422 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:412 3.fs1 3:f90c0d61:::smithi17039300-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:35:40.423 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.458230+0000", 2023-10-20T03:35:40.423 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.7483809489999995, 2023-10-20T03:35:40.423 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.042333239000000002, 2023-10-20T03:35:40.423 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.423 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:40.424 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.424 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.424 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.424 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 412 2023-10-20T03:35:40.424 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.424 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.425 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.425 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.425 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458230+0000", 2023-10-20T03:35:40.425 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.426 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.426 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.426 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.426 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458230+0000", 2023-10-20T03:35:40.426 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.426 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.427 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.427 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.427 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458231+0000", 2023-10-20T03:35:40.427 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8700000000000005e-07 2023-10-20T03:35:40.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.428 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.428 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.428 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458234+0000", 2023-10-20T03:35:40.428 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5499999999999999e-06 2023-10-20T03:35:40.428 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.429 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.429 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.429 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458245+0000", 2023-10-20T03:35:40.429 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0716999999999999e-05 2023-10-20T03:35:40.429 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.430 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.430 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.430 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458252+0000", 2023-10-20T03:35:40.430 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.486e-06 2023-10-20T03:35:40.430 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.431 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.431 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.431 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.491656+0000", 2023-10-20T03:35:40.431 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.033403571 2023-10-20T03:35:40.431 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.432 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.432 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:40.432 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.491757+0000", 2023-10-20T03:35:40.432 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101383 2023-10-20T03:35:40.432 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.432 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.433 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.433 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.498257+0000", 2023-10-20T03:35:40.433 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0064998809999999999 2023-10-20T03:35:40.433 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.433 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.434 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.434 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.498357+0000", 2023-10-20T03:35:40.434 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9491e-05 2023-10-20T03:35:40.434 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.434 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.435 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.435 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.500563+0000", 2023-10-20T03:35:40.435 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0022063730000000002 2023-10-20T03:35:40.435 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.435 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.436 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.436 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.436 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.436 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:418 3.0s0 3:0e4cfe35:::smithi17039300-49:head [append 0~770048] snapc f=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-10-20T03:35:40.437 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.456193+0000", 2023-10-20T03:35:40.437 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 3.750417949, 2023-10-20T03:35:40.437 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030331303, 2023-10-20T03:35:40.437 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.437 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:40.438 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.438 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.438 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.438 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 418 2023-10-20T03:35:40.438 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.439 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.439 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.439 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.439 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.456193+0000", 2023-10-20T03:35:40.440 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.440 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.440 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.440 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.440 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.456193+0000", 2023-10-20T03:35:40.441 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.441 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.441 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.441 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.456196+0000", 2023-10-20T03:35:40.442 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7800000000000001e-06 2023-10-20T03:35:40.442 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.442 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.442 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.442 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.457014+0000", 2023-10-20T03:35:40.443 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00081844100000000005 2023-10-20T03:35:40.443 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.443 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.443 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.443 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.457044+0000", 2023-10-20T03:35:40.444 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9737e-05 2023-10-20T03:35:40.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.444 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.444 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.445 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.457064+0000", 2023-10-20T03:35:40.445 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0032999999999999e-05 2023-10-20T03:35:40.445 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.445 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.445 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.446 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.457191+0000", 2023-10-20T03:35:40.446 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012753299999999999 2023-10-20T03:35:40.446 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.446 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.446 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.447 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.457381+0000", 2023-10-20T03:35:40.447 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000189965 2023-10-20T03:35:40.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.447 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.447 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.448 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459135+0000", 2023-10-20T03:35:40.448 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017538250000000001 2023-10-20T03:35:40.448 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.448 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.448 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.448 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.485214+0000", 2023-10-20T03:35:40.449 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026078736000000002 2023-10-20T03:35:40.449 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.449 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.449 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:40.450 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.485940+0000", 2023-10-20T03:35:40.450 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00072639799999999995 2023-10-20T03:35:40.450 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.450 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.450 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.451 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.486524+0000", 2023-10-20T03:35:40.451 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00058385499999999999 2023-10-20T03:35:40.451 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.451 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.451 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.452 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.452 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.452 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:419 3.0s0 3:0e4cfe35:::smithi17039300-49:head [append 0~475136] snapc f=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-10-20T03:35:40.452 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.464306+0000", 2023-10-20T03:35:40.453 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 3.7423045340000001, 2023-10-20T03:35:40.453 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.032538886000000003, 2023-10-20T03:35:40.453 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.453 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:40.453 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.454 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.454 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.454 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 419 2023-10-20T03:35:40.454 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.454 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.455 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.455 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.455 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.464306+0000", 2023-10-20T03:35:40.455 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.456 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.456 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.456 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.456 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.464306+0000", 2023-10-20T03:35:40.456 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.457 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.457 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.457 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.457 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.464309+0000", 2023-10-20T03:35:40.457 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0599999999999999e-06 2023-10-20T03:35:40.458 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.458 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.458 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.458 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.465080+0000", 2023-10-20T03:35:40.458 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00077099999999999998 2023-10-20T03:35:40.459 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.459 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.459 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.459 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.465123+0000", 2023-10-20T03:35:40.459 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2929000000000002e-05 2023-10-20T03:35:40.460 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.460 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.460 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.460 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.465146+0000", 2023-10-20T03:35:40.460 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2504999999999999e-05 2023-10-20T03:35:40.461 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.461 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.461 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.461 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.474013+0000", 2023-10-20T03:35:40.461 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0088671659999999992 2023-10-20T03:35:40.462 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.462 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.462 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.462 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.474203+0000", 2023-10-20T03:35:40.463 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019009100000000001 2023-10-20T03:35:40.463 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.463 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.463 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.463 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476085+0000", 2023-10-20T03:35:40.464 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001882019 2023-10-20T03:35:40.464 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.464 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.464 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.464 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.495640+0000", 2023-10-20T03:35:40.465 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019555204 2023-10-20T03:35:40.465 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.465 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.465 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:40.465 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.496443+0000", 2023-10-20T03:35:40.466 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00080256800000000003 2023-10-20T03:35:40.466 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.466 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.466 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.466 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.496845+0000", 2023-10-20T03:35:40.467 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00040234400000000001 2023-10-20T03:35:40.467 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.467 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.467 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.467 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.468 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.468 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:420 3.0s0 3:0e4cfe35:::smithi17039300-49:head [append 0~294912] snapc f=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-10-20T03:35:40.468 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.469140+0000", 2023-10-20T03:35:40.468 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 3.7374704589999999, 2023-10-20T03:35:40.468 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030769597999999999, 2023-10-20T03:35:40.469 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.469 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:40.469 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.469 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.470 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.470 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 420 2023-10-20T03:35:40.470 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.470 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.470 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.471 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.471 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469140+0000", 2023-10-20T03:35:40.471 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.471 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.471 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.472 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.472 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469140+0000", 2023-10-20T03:35:40.472 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.472 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.472 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.473 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.473 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469143+0000", 2023-10-20T03:35:40.473 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6570000000000001e-06 2023-10-20T03:35:40.473 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.473 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.474 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.474 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469630+0000", 2023-10-20T03:35:40.474 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00048736100000000001 2023-10-20T03:35:40.474 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.475 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.475 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.475 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469670+0000", 2023-10-20T03:35:40.475 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9712000000000002e-05 2023-10-20T03:35:40.475 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.476 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.476 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.476 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469693+0000", 2023-10-20T03:35:40.476 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2616000000000002e-05 2023-10-20T03:35:40.476 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.477 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.477 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.477 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.486686+0000", 2023-10-20T03:35:40.477 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016993774999999999 2023-10-20T03:35:40.477 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.478 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.478 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.478 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.486867+0000", 2023-10-20T03:35:40.478 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000180595 2023-10-20T03:35:40.478 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.479 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.479 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.479 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488131+0000", 2023-10-20T03:35:40.479 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0012640959999999999 2023-10-20T03:35:40.479 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.480 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.480 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.480 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.499384+0000", 2023-10-20T03:35:40.480 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011252669999999999 2023-10-20T03:35:40.480 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.481 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.481 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:40.481 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.499568+0000", 2023-10-20T03:35:40.481 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000183802 2023-10-20T03:35:40.482 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.482 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.482 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.482 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.499910+0000", 2023-10-20T03:35:40.482 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00034231399999999998 2023-10-20T03:35:40.483 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.483 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.483 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.483 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.483 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.484 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:421 3.0s0 3:0e4cfe35:::smithi17039300-49:head [setxattr _header (57) in=7b] snapc f=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-10-20T03:35:40.484 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.469730+0000", 2023-10-20T03:35:40.484 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 3.7368811709999998, 2023-10-20T03:35:40.484 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.033931731999999999, 2023-10-20T03:35:40.484 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.485 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:40.485 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.485 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.485 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.485 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 421 2023-10-20T03:35:40.486 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.486 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.486 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.486 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.486 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469730+0000", 2023-10-20T03:35:40.487 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.487 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.487 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.487 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.487 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469730+0000", 2023-10-20T03:35:40.488 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.488 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.488 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.488 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.488 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469731+0000", 2023-10-20T03:35:40.489 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5799999999999999e-06 2023-10-20T03:35:40.489 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.489 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.489 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.489 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469741+0000", 2023-10-20T03:35:40.490 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6700000000000006e-06 2023-10-20T03:35:40.490 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.490 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.490 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.491 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469765+0000", 2023-10-20T03:35:40.491 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4216e-05 2023-10-20T03:35:40.491 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.491 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.491 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.492 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469780+0000", 2023-10-20T03:35:40.492 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.482e-05 2023-10-20T03:35:40.492 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.492 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.492 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.493 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.496944+0000", 2023-10-20T03:35:40.493 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027164562999999999 2023-10-20T03:35:40.493 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.493 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.493 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.494 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.497153+0000", 2023-10-20T03:35:40.494 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020828900000000001 2023-10-20T03:35:40.494 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.494 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.495 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.495 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.498227+0000", 2023-10-20T03:35:40.495 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0010742670000000001 2023-10-20T03:35:40.495 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.495 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.496 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.496 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.501137+0000", 2023-10-20T03:35:40.496 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0029095969999999999 2023-10-20T03:35:40.496 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.496 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.497 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:40.497 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.503446+0000", 2023-10-20T03:35:40.497 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0023098580000000001 2023-10-20T03:35:40.497 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.497 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.498 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.498 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.503661+0000", 2023-10-20T03:35:40.498 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00021487200000000001 2023-10-20T03:35:40.498 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.498 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.499 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.499 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.499 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.499 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:422 3.0s0 3:0e4cfe35:::smithi17039300-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2023-10-20T03:35:40.500 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.469801+0000", 2023-10-20T03:35:40.500 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 3.7368101330000001, 2023-10-20T03:35:40.500 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.036242191, 2023-10-20T03:35:40.500 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.500 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:40.501 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:40.501 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:40.501 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:40.501 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 422 2023-10-20T03:35:40.502 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.502 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.502 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.502 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.502 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469801+0000", 2023-10-20T03:35:40.503 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.503 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.503 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.503 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.503 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469801+0000", 2023-10-20T03:35:40.504 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.504 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.504 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.504 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.504 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469802+0000", 2023-10-20T03:35:40.505 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.19e-06 2023-10-20T03:35:40.505 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.505 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.505 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.506 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469809+0000", 2023-10-20T03:35:40.506 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4630000000000004e-06 2023-10-20T03:35:40.506 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.506 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.506 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.507 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469832+0000", 2023-10-20T03:35:40.507 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2883000000000001e-05 2023-10-20T03:35:40.507 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.507 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.507 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.508 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469846+0000", 2023-10-20T03:35:40.508 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4113e-05 2023-10-20T03:35:40.508 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.508 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.508 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.509 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.500041+0000", 2023-10-20T03:35:40.509 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030195161000000002 2023-10-20T03:35:40.509 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.509 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.509 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:40.510 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.500396+0000", 2023-10-20T03:35:40.510 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00035421399999999999 2023-10-20T03:35:40.510 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.510 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.510 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.511 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.504208+0000", 2023-10-20T03:35:40.511 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0038123390000000001 2023-10-20T03:35:40.511 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.511 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.511 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:40.512 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.504283+0000", 2023-10-20T03:35:40.512 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4981999999999995e-05 2023-10-20T03:35:40.512 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.512 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.512 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.513 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.506043+0000", 2023-10-20T03:35:40.513 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001759846 2023-10-20T03:35:40.513 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.513 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.514 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.514 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.514 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.514 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:40.515 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15] out_osds: [12] dead_osds: [6] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 10, 14, 3, 7, 11, 15] 2023-10-20T03:35:40.515 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-10-20T03:35:40.515 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2023-10-20T03:35:40.516 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-10-20T03:35:40.516 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-10-20T03:35:40.516 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2023-10-20T03:35:40.518 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:40.512+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:40.519 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-10-20T03:35:40.617 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:40.613+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:40.687 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:40.688 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:40.688 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2023-10-20T03:35:40.688 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:40.696 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:40.696 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-10-20T03:35:40.696 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:40.702 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-10-20T03:35:40.718 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:40.716+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:40.813 INFO:tasks.rados.rados.0.smithi170.stdout:186: done (3 left) 2023-10-20T03:35:40.813 INFO:tasks.rados.rados.0.smithi170.stdout:187: done (2 left) 2023-10-20T03:35:40.813 INFO:tasks.rados.rados.0.smithi170.stdout:188: done (1 left) 2023-10-20T03:35:40.813 INFO:tasks.rados.rados.0.smithi170.stdout:190: done (0 left) 2023-10-20T03:35:40.814 INFO:tasks.rados.rados.0.smithi170.stdout:191: snap_remove snap 1 2023-10-20T03:35:40.818 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:40.817+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:40.821 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-10-20T03:35:40.823 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:40.823 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:40.823 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:35:40.824 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:35:40.824 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:40.837 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-10-20T03:35:40.919 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:40.917+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:40.961 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:40.961 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2023-10-20T03:35:40.961 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2023-10-20T03:35:40.962 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2023-10-20T03:35:40.962 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.962 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.2s6 36/31 ECSubWrite(tid=21, reqid=client.4570.0:198, at_version=36'13, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:40.962 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:18.266756+0000", 2023-10-20T03:35:40.962 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.692278510000001, 2023-10-20T03:35:40.963 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025327286000000001, 2023-10-20T03:35:40.963 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.963 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:40.963 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.963 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.964 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.964 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.266756+0000", 2023-10-20T03:35:40.964 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.964 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.964 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.965 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.965 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.266756+0000", 2023-10-20T03:35:40.965 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.965 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.965 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.965 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.966 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.266756+0000", 2023-10-20T03:35:40.966 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4299999999999999e-07 2023-10-20T03:35:40.966 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.966 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.966 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.967 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.267218+0000", 2023-10-20T03:35:40.967 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00046247200000000001 2023-10-20T03:35:40.967 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.967 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.967 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.968 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.267257+0000", 2023-10-20T03:35:40.968 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8973000000000001e-05 2023-10-20T03:35:40.968 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.968 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.968 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.968 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.267276+0000", 2023-10-20T03:35:40.969 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8434999999999999e-05 2023-10-20T03:35:40.969 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.969 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.969 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.969 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.267440+0000", 2023-10-20T03:35:40.969 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000164081 2023-10-20T03:35:40.970 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.970 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.970 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.970 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.267467+0000", 2023-10-20T03:35:40.970 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7512e-05 2023-10-20T03:35:40.971 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.971 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.971 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.971 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.291988+0000", 2023-10-20T03:35:40.971 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.024520897 2023-10-20T03:35:40.972 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.972 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.972 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.972 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.292083+0000", 2023-10-20T03:35:40.972 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4573000000000004e-05 2023-10-20T03:35:40.973 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.973 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.973 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.973 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.973 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.974 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.2s6 36/31 ECSubWrite(tid=22, reqid=client.4570.0:199, at_version=36'14, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:40.974 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:18.284375+0000", 2023-10-20T03:35:40.974 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.674658696000002, 2023-10-20T03:35:40.974 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0084027949999999994, 2023-10-20T03:35:40.974 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.975 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:40.975 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.975 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.975 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.975 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.284375+0000", 2023-10-20T03:35:40.976 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.976 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.976 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.976 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.976 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.284375+0000", 2023-10-20T03:35:40.976 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.977 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.977 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.977 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.977 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.284377+0000", 2023-10-20T03:35:40.977 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.103e-06 2023-10-20T03:35:40.978 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.978 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.978 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.978 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.284599+0000", 2023-10-20T03:35:40.978 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00022122299999999999 2023-10-20T03:35:40.979 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.979 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.979 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.979 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.284633+0000", 2023-10-20T03:35:40.979 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4252999999999999e-05 2023-10-20T03:35:40.980 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.980 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.980 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.980 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.284655+0000", 2023-10-20T03:35:40.980 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2347000000000001e-05 2023-10-20T03:35:40.981 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.981 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.981 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.981 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.284941+0000", 2023-10-20T03:35:40.981 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000286058 2023-10-20T03:35:40.981 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.982 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.982 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.982 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.284958+0000", 2023-10-20T03:35:40.982 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7125e-05 2023-10-20T03:35:40.982 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.983 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.983 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.983 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.292697+0000", 2023-10-20T03:35:40.983 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0077388960000000003 2023-10-20T03:35:40.983 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.984 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.984 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.984 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.292778+0000", 2023-10-20T03:35:40.984 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0790000000000004e-05 2023-10-20T03:35:40.984 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.984 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.985 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.985 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.985 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.985 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.1s4 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:40.986 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:25.329817+0000", 2023-10-20T03:35:40.986 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 15.629216544, 2023-10-20T03:35:40.986 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026006504999999999, 2023-10-20T03:35:40.986 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.986 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:40.986 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.987 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.987 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.987 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.329817+0000", 2023-10-20T03:35:40.987 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.988 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.988 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.988 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:40.988 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.329817+0000", 2023-10-20T03:35:40.988 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.989 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.989 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.989 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:40.989 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.329818+0000", 2023-10-20T03:35:40.989 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0699999999999997e-07 2023-10-20T03:35:40.990 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.990 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.990 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:40.990 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330431+0000", 2023-10-20T03:35:40.990 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00061258999999999999 2023-10-20T03:35:40.991 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.991 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.991 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:40.991 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330468+0000", 2023-10-20T03:35:40.991 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7651999999999998e-05 2023-10-20T03:35:40.991 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.992 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.992 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:40.992 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330493+0000", 2023-10-20T03:35:40.992 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4766e-05 2023-10-20T03:35:40.992 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.993 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.993 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:40.993 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330710+0000", 2023-10-20T03:35:40.993 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00021685000000000001 2023-10-20T03:35:40.993 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.993 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.994 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:40.994 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330742+0000", 2023-10-20T03:35:40.994 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2406000000000001e-05 2023-10-20T03:35:40.994 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.994 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.995 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:40.995 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.355705+0000", 2023-10-20T03:35:40.995 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.024962682 2023-10-20T03:35:40.995 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.995 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.996 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:40.996 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.355824+0000", 2023-10-20T03:35:40.996 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000118952 2023-10-20T03:35:40.996 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.996 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:40.997 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:40.997 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.997 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.997 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.1s4 43/31 ECSubWrite(tid=81, reqid=client.4570.0:307, at_version=43'15, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:40.997 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:25.343566+0000", 2023-10-20T03:35:40.997 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 15.615467576, 2023-10-20T03:35:40.998 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015876868999999998, 2023-10-20T03:35:40.998 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:40.998 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:40.998 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:40.998 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:40.999 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:40.999 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.343566+0000", 2023-10-20T03:35:40.999 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:40.999 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:40.999 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.000 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.000 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.343566+0000", 2023-10-20T03:35:41.000 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.000 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.000 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.000 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.001 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.343569+0000", 2023-10-20T03:35:41.001 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3829999999999999e-06 2023-10-20T03:35:41.001 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.001 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.001 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.002 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.343931+0000", 2023-10-20T03:35:41.002 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00036177099999999999 2023-10-20T03:35:41.002 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.002 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.002 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.003 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.343959+0000", 2023-10-20T03:35:41.003 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8663e-05 2023-10-20T03:35:41.003 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.003 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.004 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.004 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.343980+0000", 2023-10-20T03:35:41.004 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0282999999999999e-05 2023-10-20T03:35:41.004 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.004 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.005 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.005 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.344207+0000", 2023-10-20T03:35:41.005 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00022750900000000001 2023-10-20T03:35:41.005 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.005 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.006 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.006 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.344249+0000", 2023-10-20T03:35:41.006 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1659000000000002e-05 2023-10-20T03:35:41.006 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.006 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.006 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.007 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.359360+0000", 2023-10-20T03:35:41.007 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015111253999999999 2023-10-20T03:35:41.007 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.007 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.007 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.007 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.359443+0000", 2023-10-20T03:35:41.008 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3347e-05 2023-10-20T03:35:41.008 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.008 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.008 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.008 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.009 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.009 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.1s4 43/31 ECSubWrite(tid=83, reqid=client.4570.0:308, at_version=43'16, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:41.009 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:25.356466+0000", 2023-10-20T03:35:41.009 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 15.602567860000001, 2023-10-20T03:35:41.009 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0052969829999999999, 2023-10-20T03:35:41.009 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.010 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.010 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.010 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.010 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.010 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.356466+0000", 2023-10-20T03:35:41.010 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.011 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.011 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.011 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.011 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.356466+0000", 2023-10-20T03:35:41.011 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.012 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.012 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.012 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.012 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.356468+0000", 2023-10-20T03:35:41.012 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.905e-06 2023-10-20T03:35:41.013 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.013 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.013 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.013 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.356587+0000", 2023-10-20T03:35:41.013 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000119164 2023-10-20T03:35:41.014 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.014 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.014 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.014 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.356624+0000", 2023-10-20T03:35:41.014 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6303999999999997e-05 2023-10-20T03:35:41.015 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.015 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.015 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.015 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.356643+0000", 2023-10-20T03:35:41.015 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9837e-05 2023-10-20T03:35:41.015 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.016 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.016 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.016 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.356838+0000", 2023-10-20T03:35:41.016 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019461899999999999 2023-10-20T03:35:41.016 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.017 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.017 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.017 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.356865+0000", 2023-10-20T03:35:41.017 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6832999999999999e-05 2023-10-20T03:35:41.017 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.018 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.018 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.018 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.361653+0000", 2023-10-20T03:35:41.018 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0047878410000000001 2023-10-20T03:35:41.018 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.019 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.019 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.019 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.361763+0000", 2023-10-20T03:35:41.019 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011048 2023-10-20T03:35:41.019 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.020 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.020 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.020 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.020 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.020 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s1 44/31 ECSubWrite(tid=18, reqid=client.4570.0:323, at_version=44'9, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:35:41.021 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.361569+0000", 2023-10-20T03:35:41.021 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.597465193, 2023-10-20T03:35:41.021 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019257209000000001, 2023-10-20T03:35:41.021 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.021 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.022 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.022 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.022 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.022 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361569+0000", 2023-10-20T03:35:41.022 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.022 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.023 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.023 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.023 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361569+0000", 2023-10-20T03:35:41.023 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.023 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.024 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.024 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.024 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361571+0000", 2023-10-20T03:35:41.024 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4370000000000001e-06 2023-10-20T03:35:41.024 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.025 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.025 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.025 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361886+0000", 2023-10-20T03:35:41.025 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00031429 2023-10-20T03:35:41.025 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.026 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.026 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.026 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361927+0000", 2023-10-20T03:35:41.026 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1174999999999998e-05 2023-10-20T03:35:41.026 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.027 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.027 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.027 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361950+0000", 2023-10-20T03:35:41.027 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3048999999999999e-05 2023-10-20T03:35:41.027 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.028 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.028 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.028 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.362128+0000", 2023-10-20T03:35:41.028 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017819400000000001 2023-10-20T03:35:41.028 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.028 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.029 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.029 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.362154+0000", 2023-10-20T03:35:41.029 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6049000000000001e-05 2023-10-20T03:35:41.029 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.029 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.030 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.030 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.380718+0000", 2023-10-20T03:35:41.030 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.018564051000000002 2023-10-20T03:35:41.030 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.030 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.031 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.031 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.380826+0000", 2023-10-20T03:35:41.031 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000107964 2023-10-20T03:35:41.031 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.031 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.032 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.032 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.032 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.032 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s1 44/31 ECSubWrite(tid=19, reqid=client.4570.0:324, at_version=44'10, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:35:41.032 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.388436+0000", 2023-10-20T03:35:41.032 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.570598438999999, 2023-10-20T03:35:41.033 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014334856, 2023-10-20T03:35:41.033 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.033 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.033 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.033 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.034 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.034 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388436+0000", 2023-10-20T03:35:41.034 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.034 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.034 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.035 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.035 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388436+0000", 2023-10-20T03:35:41.035 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.035 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.035 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.036 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.036 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388437+0000", 2023-10-20T03:35:41.036 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5060000000000001e-06 2023-10-20T03:35:41.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.037 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.037 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388568+0000", 2023-10-20T03:35:41.037 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013096599999999999 2023-10-20T03:35:41.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.037 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.037 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.038 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388594+0000", 2023-10-20T03:35:41.038 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6129999999999999e-05 2023-10-20T03:35:41.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.038 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.038 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.038 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388608+0000", 2023-10-20T03:35:41.039 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4133e-05 2023-10-20T03:35:41.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.039 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.039 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.039 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388766+0000", 2023-10-20T03:35:41.040 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015721100000000001 2023-10-20T03:35:41.040 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.040 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.040 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.040 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388793+0000", 2023-10-20T03:35:41.040 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7699e-05 2023-10-20T03:35:41.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.041 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.041 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.041 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402725+0000", 2023-10-20T03:35:41.041 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013931294 2023-10-20T03:35:41.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.042 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.042 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.042 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402770+0000", 2023-10-20T03:35:41.042 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5917000000000001e-05 2023-10-20T03:35:41.042 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.043 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.043 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.043 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.043 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.043 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s1 44/31 ECSubWrite(tid=20, reqid=client.4570.0:325, at_version=44'11, trim_to=0'0, roll_forward_to=44'9))", 2023-10-20T03:35:41.044 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.398972+0000", 2023-10-20T03:35:41.044 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.560061859999999, 2023-10-20T03:35:41.044 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.004981063, 2023-10-20T03:35:41.044 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.044 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.044 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.045 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.045 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.045 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.398972+0000", 2023-10-20T03:35:41.045 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.046 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.046 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.046 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.398972+0000", 2023-10-20T03:35:41.046 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.046 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.047 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.047 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.398974+0000", 2023-10-20T03:35:41.047 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9369999999999998e-06 2023-10-20T03:35:41.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.047 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.048 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.048 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.399093+0000", 2023-10-20T03:35:41.048 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000118596 2023-10-20T03:35:41.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.048 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.049 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.399157+0000", 2023-10-20T03:35:41.049 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3978000000000003e-05 2023-10-20T03:35:41.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.049 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.049 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.050 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.399183+0000", 2023-10-20T03:35:41.050 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6786000000000001e-05 2023-10-20T03:35:41.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.050 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.050 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.051 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.399362+0000", 2023-10-20T03:35:41.051 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000178383 2023-10-20T03:35:41.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.051 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.051 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.051 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.399384+0000", 2023-10-20T03:35:41.052 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1690000000000001e-05 2023-10-20T03:35:41.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.052 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.052 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.052 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.403896+0000", 2023-10-20T03:35:41.053 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0045125030000000002 2023-10-20T03:35:41.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.053 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.053 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.053 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.403953+0000", 2023-10-20T03:35:41.054 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.719e-05 2023-10-20T03:35:41.054 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.054 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.054 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.054 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.055 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.1s4 44/31 ECSubWrite(tid=97, reqid=client.4570.0:341, at_version=44'19, trim_to=0'0, roll_forward_to=43'17))", 2023-10-20T03:35:41.055 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.402679+0000", 2023-10-20T03:35:41.055 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.556355057999999, 2023-10-20T03:35:41.055 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0056050709999999997, 2023-10-20T03:35:41.055 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.056 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.056 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.056 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.056 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.056 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402679+0000", 2023-10-20T03:35:41.057 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.057 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.057 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.057 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402679+0000", 2023-10-20T03:35:41.058 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.058 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.058 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.058 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.058 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402680+0000", 2023-10-20T03:35:41.059 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9699999999999996e-07 2023-10-20T03:35:41.059 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.059 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.059 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.059 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402730+0000", 2023-10-20T03:35:41.059 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0711999999999998e-05 2023-10-20T03:35:41.060 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.060 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.060 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.060 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402764+0000", 2023-10-20T03:35:41.060 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3451999999999998e-05 2023-10-20T03:35:41.061 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.061 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.061 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.061 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402786+0000", 2023-10-20T03:35:41.061 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1909e-05 2023-10-20T03:35:41.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.062 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.062 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.062 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.403057+0000", 2023-10-20T03:35:41.062 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00027106199999999997 2023-10-20T03:35:41.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.063 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.063 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.063 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.403125+0000", 2023-10-20T03:35:41.063 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8001000000000002e-05 2023-10-20T03:35:41.063 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.064 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.064 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.064 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.408245+0000", 2023-10-20T03:35:41.064 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0051203689999999996 2023-10-20T03:35:41.064 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.064 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.065 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.065 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.408284+0000", 2023-10-20T03:35:41.065 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8968999999999999e-05 2023-10-20T03:35:41.065 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.065 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.065 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.066 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.066 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.066 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:343 3.9s0 3:93433de7:::smithi17039300-36:head [read 0~1818624,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:41.066 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.409430+0000", 2023-10-20T03:35:41.066 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.549603831000001, 2023-10-20T03:35:41.067 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0079541439999999998, 2023-10-20T03:35:41.067 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.067 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:41.067 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:41.067 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:41.067 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 343 2023-10-20T03:35:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.068 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409430+0000", 2023-10-20T03:35:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409430+0000", 2023-10-20T03:35:41.069 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409433+0000", 2023-10-20T03:35:41.070 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6000000000000001e-06 2023-10-20T03:35:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409444+0000", 2023-10-20T03:35:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0947e-05 2023-10-20T03:35:41.071 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409478+0000", 2023-10-20T03:35:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4227000000000003e-05 2023-10-20T03:35:41.072 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409500+0000", 2023-10-20T03:35:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2316000000000001e-05 2023-10-20T03:35:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.073 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.074 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.074 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409605+0000", 2023-10-20T03:35:41.074 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000104417 2023-10-20T03:35:41.074 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.074 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.075 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:41.075 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409722+0000", 2023-10-20T03:35:41.075 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011777699999999999 2023-10-20T03:35:41.075 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.075 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.075 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.076 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.417384+0000", 2023-10-20T03:35:41.076 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0076618600000000004 2023-10-20T03:35:41.076 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.076 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.076 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.076 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.077 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.077 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:344 3.9s0 3:93433de7:::smithi17039300-36:head [read 0~1818624,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:41.077 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.409542+0000", 2023-10-20T03:35:41.077 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.549492462, 2023-10-20T03:35:41.077 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0095144679999999999, 2023-10-20T03:35:41.078 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.078 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:41.078 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:41.078 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:41.078 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:41.079 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 344 2023-10-20T03:35:41.079 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.079 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.079 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.079 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.080 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409542+0000", 2023-10-20T03:35:41.080 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.080 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.080 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.080 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.080 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409542+0000", 2023-10-20T03:35:41.081 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.081 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.081 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.081 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.081 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409543+0000", 2023-10-20T03:35:41.082 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0750000000000001e-06 2023-10-20T03:35:41.082 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.082 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.082 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.082 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409550+0000", 2023-10-20T03:35:41.083 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2269999999999996e-06 2023-10-20T03:35:41.083 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.083 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.083 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.084 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409567+0000", 2023-10-20T03:35:41.084 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.734e-05 2023-10-20T03:35:41.084 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.084 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.084 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.085 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409592+0000", 2023-10-20T03:35:41.085 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5037000000000001e-05 2023-10-20T03:35:41.085 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.085 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.085 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.086 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.410630+0000", 2023-10-20T03:35:41.086 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0010379300000000001 2023-10-20T03:35:41.086 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.086 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.086 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:41.086 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.410754+0000", 2023-10-20T03:35:41.087 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012393899999999999 2023-10-20T03:35:41.087 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.087 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.087 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.087 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.419056+0000", 2023-10-20T03:35:41.088 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0083019200000000008 2023-10-20T03:35:41.088 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.088 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.088 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.088 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.089 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.089 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:345 3.9s0 3:93433de7:::smithi17039300-36:head [read 0~1818624,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:41.089 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.409618+0000", 2023-10-20T03:35:41.089 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.549416494000001, 2023-10-20T03:35:41.089 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010854007000000001, 2023-10-20T03:35:41.089 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.090 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:41.090 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:41.090 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:41.090 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:41.090 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 345 2023-10-20T03:35:41.091 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.091 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.091 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.091 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.091 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409618+0000", 2023-10-20T03:35:41.092 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.092 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.092 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.092 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.092 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409618+0000", 2023-10-20T03:35:41.092 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.093 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.093 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.093 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.093 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409619+0000", 2023-10-20T03:35:41.093 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.850000000000001e-07 2023-10-20T03:35:41.093 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.094 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.094 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.094 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409624+0000", 2023-10-20T03:35:41.094 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4070000000000001e-06 2023-10-20T03:35:41.094 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.095 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.095 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.095 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409639+0000", 2023-10-20T03:35:41.095 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5359999999999999e-05 2023-10-20T03:35:41.095 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.095 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.096 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.096 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.409676+0000", 2023-10-20T03:35:41.096 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6582999999999999e-05 2023-10-20T03:35:41.096 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.096 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.097 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.097 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.412343+0000", 2023-10-20T03:35:41.097 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0026672480000000001 2023-10-20T03:35:41.097 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.097 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.098 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:41.098 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.412489+0000", 2023-10-20T03:35:41.098 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014556000000000001 2023-10-20T03:35:41.098 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.098 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.099 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.099 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.420472+0000", 2023-10-20T03:35:41.099 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0079828639999999992 2023-10-20T03:35:41.099 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.099 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.099 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.100 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.100 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.100 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.2s6 49/31 ECSubWrite(tid=34, reqid=osd.14.0:34, at_version=49'19, trim_to=0'0, roll_forward_to=44'17))", 2023-10-20T03:35:41.100 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:31.395243+0000", 2023-10-20T03:35:41.100 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 9.5637908419999995, 2023-10-20T03:35:41.100 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0063105970000000003, 2023-10-20T03:35:41.101 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.101 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.101 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.101 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.101 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.102 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.395243+0000", 2023-10-20T03:35:41.102 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.102 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.102 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.102 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.102 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.395243+0000", 2023-10-20T03:35:41.103 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.103 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.103 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.103 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.103 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.395244+0000", 2023-10-20T03:35:41.104 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4300000000000003e-07 2023-10-20T03:35:41.104 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.104 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.104 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.104 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.395305+0000", 2023-10-20T03:35:41.105 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1005000000000003e-05 2023-10-20T03:35:41.105 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.105 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.105 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.105 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.395331+0000", 2023-10-20T03:35:41.106 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5834999999999999e-05 2023-10-20T03:35:41.106 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.106 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.106 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.106 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.395353+0000", 2023-10-20T03:35:41.107 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2407000000000001e-05 2023-10-20T03:35:41.107 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.107 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.107 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.107 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.396011+0000", 2023-10-20T03:35:41.107 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00065766499999999999 2023-10-20T03:35:41.108 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.108 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.108 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.108 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.396036+0000", 2023-10-20T03:35:41.108 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5496000000000001e-05 2023-10-20T03:35:41.109 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.109 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.109 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.109 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.401408+0000", 2023-10-20T03:35:41.109 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0053723369999999996 2023-10-20T03:35:41.110 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.110 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.110 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.110 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.401554+0000", 2023-10-20T03:35:41.110 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014530899999999999 2023-10-20T03:35:41.111 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.111 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.111 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.111 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.111 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.112 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.1s4 49/31 ECSubWrite(tid=117, reqid=osd.13.0:117, at_version=49'22, trim_to=0'0, roll_forward_to=44'19))", 2023-10-20T03:35:41.112 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:31.404845+0000", 2023-10-20T03:35:41.112 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 9.5541888830000001, 2023-10-20T03:35:41.112 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00507654, 2023-10-20T03:35:41.112 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.112 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.113 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.113 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.113 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.113 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.404845+0000", 2023-10-20T03:35:41.113 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.113 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.114 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.114 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.114 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.404845+0000", 2023-10-20T03:35:41.114 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.114 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.115 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.115 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.115 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.404846+0000", 2023-10-20T03:35:41.115 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2399999999999998e-07 2023-10-20T03:35:41.115 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.116 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.116 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.116 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.404902+0000", 2023-10-20T03:35:41.116 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6505000000000002e-05 2023-10-20T03:35:41.116 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.117 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.117 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.117 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.404934+0000", 2023-10-20T03:35:41.117 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2066000000000001e-05 2023-10-20T03:35:41.117 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.118 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.118 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.118 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.404958+0000", 2023-10-20T03:35:41.118 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3561999999999999e-05 2023-10-20T03:35:41.118 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.118 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.119 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.119 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.405916+0000", 2023-10-20T03:35:41.119 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00095859000000000005 2023-10-20T03:35:41.119 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.119 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.119 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.120 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.405943+0000", 2023-10-20T03:35:41.120 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6137000000000001e-05 2023-10-20T03:35:41.120 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.120 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.120 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.120 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.409826+0000", 2023-10-20T03:35:41.121 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0038838919999999999 2023-10-20T03:35:41.121 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.121 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.121 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.121 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.409922+0000", 2023-10-20T03:35:41.121 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5264000000000001e-05 2023-10-20T03:35:41.122 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.122 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.122 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.122 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.122 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.123 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.cs4 49/47 ECSubWrite(tid=118, reqid=osd.13.0:118, at_version=49'29, trim_to=0'0, roll_forward_to=48'27))", 2023-10-20T03:35:41.123 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:31.406524+0000", 2023-10-20T03:35:41.123 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 9.5525104620000008, 2023-10-20T03:35:41.123 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0054851880000000002, 2023-10-20T03:35:41.123 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.123 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.124 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.124 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.124 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.124 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.406524+0000", 2023-10-20T03:35:41.124 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.124 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.125 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.125 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.125 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.406524+0000", 2023-10-20T03:35:41.125 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.125 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.126 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.126 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.126 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.406525+0000", 2023-10-20T03:35:41.126 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5960000000000001e-06 2023-10-20T03:35:41.127 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.127 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.127 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.127 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.406574+0000", 2023-10-20T03:35:41.127 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9024999999999999e-05 2023-10-20T03:35:41.128 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.128 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.128 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.128 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.406601+0000", 2023-10-20T03:35:41.128 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7010000000000001e-05 2023-10-20T03:35:41.129 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.129 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.129 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.129 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.406629+0000", 2023-10-20T03:35:41.129 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7889e-05 2023-10-20T03:35:41.129 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.130 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.130 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.130 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.406828+0000", 2023-10-20T03:35:41.130 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019851800000000001 2023-10-20T03:35:41.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.131 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.131 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.131 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.406852+0000", 2023-10-20T03:35:41.131 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4723e-05 2023-10-20T03:35:41.132 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.132 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.132 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.132 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.411909+0000", 2023-10-20T03:35:41.132 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0050563989999999996 2023-10-20T03:35:41.132 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.133 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.133 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.133 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.412009+0000", 2023-10-20T03:35:41.133 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000100028 2023-10-20T03:35:41.133 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.134 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.134 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.134 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.134 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.134 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s1 49/31 ECSubWrite(tid=39, reqid=client.4570.0:390, at_version=49'23, trim_to=0'0, roll_forward_to=47'21))", 2023-10-20T03:35:41.134 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:31.413927+0000", 2023-10-20T03:35:41.135 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 9.5451070409999996, 2023-10-20T03:35:41.135 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026987964999999999, 2023-10-20T03:35:41.135 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.135 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.135 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.135 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.136 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.136 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.413927+0000", 2023-10-20T03:35:41.136 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.136 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.136 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.137 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.137 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.413927+0000", 2023-10-20T03:35:41.137 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.137 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.137 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.137 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.138 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.413928+0000", 2023-10-20T03:35:41.138 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5700000000000001e-07 2023-10-20T03:35:41.138 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.138 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.138 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.139 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.414488+0000", 2023-10-20T03:35:41.139 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00056035099999999995 2023-10-20T03:35:41.139 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.139 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.139 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.140 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.414523+0000", 2023-10-20T03:35:41.140 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5262000000000002e-05 2023-10-20T03:35:41.140 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.140 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.140 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.140 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.414546+0000", 2023-10-20T03:35:41.141 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2860000000000001e-05 2023-10-20T03:35:41.141 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.141 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.141 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.141 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.414713+0000", 2023-10-20T03:35:41.142 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016685199999999999 2023-10-20T03:35:41.142 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.142 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.142 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.143 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.414739+0000", 2023-10-20T03:35:41.143 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5721999999999999e-05 2023-10-20T03:35:41.143 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.143 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.143 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.144 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.440829+0000", 2023-10-20T03:35:41.144 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026089833999999999 2023-10-20T03:35:41.144 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.144 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.144 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.145 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.440915+0000", 2023-10-20T03:35:41.145 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6227e-05 2023-10-20T03:35:41.145 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.145 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.145 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.145 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.146 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.146 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.2s6 53/31 ECSubWrite(tid=40, reqid=osd.14.0:40, at_version=53'25, trim_to=0'0, roll_forward_to=51'23))", 2023-10-20T03:35:41.146 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.429367+0000", 2023-10-20T03:35:41.146 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 5.5296670849999998, 2023-10-20T03:35:41.146 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0053658040000000001, 2023-10-20T03:35:41.147 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.147 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.147 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.147 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.147 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.147 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.429367+0000", 2023-10-20T03:35:41.148 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.148 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.148 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.148 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.148 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.429367+0000", 2023-10-20T03:35:41.149 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.149 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.149 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.149 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.149 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.429368+0000", 2023-10-20T03:35:41.150 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2300000000000001e-07 2023-10-20T03:35:41.150 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.150 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.150 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.150 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.429418+0000", 2023-10-20T03:35:41.150 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0285999999999999e-05 2023-10-20T03:35:41.151 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.151 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.151 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.151 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.429442+0000", 2023-10-20T03:35:41.151 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4108999999999999e-05 2023-10-20T03:35:41.151 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.152 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.152 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.152 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.429467+0000", 2023-10-20T03:35:41.152 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5066e-05 2023-10-20T03:35:41.152 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.153 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.153 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.153 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.429652+0000", 2023-10-20T03:35:41.153 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018484600000000001 2023-10-20T03:35:41.153 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.153 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.154 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.154 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.429708+0000", 2023-10-20T03:35:41.154 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5708000000000003e-05 2023-10-20T03:35:41.154 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.154 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.155 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.155 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.434641+0000", 2023-10-20T03:35:41.155 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0049337950000000004 2023-10-20T03:35:41.155 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.155 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.156 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.156 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.434733+0000", 2023-10-20T03:35:41.156 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1371000000000003e-05 2023-10-20T03:35:41.156 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.156 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.157 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.157 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.157 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.157 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s1 54/31 ECSubWrite(tid=65, reqid=client.4570.0:418, at_version=54'32, trim_to=0'0, roll_forward_to=51'31))", 2023-10-20T03:35:41.157 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.459265+0000", 2023-10-20T03:35:41.157 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.4997693830000003, 2023-10-20T03:35:41.158 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016601141, 2023-10-20T03:35:41.158 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.158 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.158 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.158 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.159 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.159 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459265+0000", 2023-10-20T03:35:41.159 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.159 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.159 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.160 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.160 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459265+0000", 2023-10-20T03:35:41.160 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.160 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.160 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.161 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.161 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459265+0000", 2023-10-20T03:35:41.161 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3499999999999996e-07 2023-10-20T03:35:41.161 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.161 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.162 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.162 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459575+0000", 2023-10-20T03:35:41.162 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00030992299999999998 2023-10-20T03:35:41.162 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.162 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.163 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.163 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459606+0000", 2023-10-20T03:35:41.163 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0451e-05 2023-10-20T03:35:41.163 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.163 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.164 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.164 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459628+0000", 2023-10-20T03:35:41.164 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2248000000000001e-05 2023-10-20T03:35:41.164 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.164 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.165 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.165 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459783+0000", 2023-10-20T03:35:41.165 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015505099999999999 2023-10-20T03:35:41.165 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.165 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.166 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.166 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459810+0000", 2023-10-20T03:35:41.166 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7061999999999999e-05 2023-10-20T03:35:41.166 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.166 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.167 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.167 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.475760+0000", 2023-10-20T03:35:41.167 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015950496000000002 2023-10-20T03:35:41.167 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.167 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.167 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.168 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.475866+0000", 2023-10-20T03:35:41.168 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000105375 2023-10-20T03:35:41.168 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.168 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.168 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.169 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.169 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.169 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s1 54/31 ECSubWrite(tid=66, reqid=client.4570.0:419, at_version=54'33, trim_to=0'0, roll_forward_to=51'31))", 2023-10-20T03:35:41.169 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.476224+0000", 2023-10-20T03:35:41.169 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.482810336, 2023-10-20T03:35:41.170 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011431317, 2023-10-20T03:35:41.170 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.170 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.170 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.170 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.171 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.171 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476224+0000", 2023-10-20T03:35:41.171 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.171 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.171 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.171 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.172 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476224+0000", 2023-10-20T03:35:41.172 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.172 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.172 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.172 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.173 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476226+0000", 2023-10-20T03:35:41.173 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2000000000000001e-06 2023-10-20T03:35:41.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.173 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.173 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.174 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476470+0000", 2023-10-20T03:35:41.174 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024458800000000001 2023-10-20T03:35:41.174 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.174 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.174 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.175 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476500+0000", 2023-10-20T03:35:41.175 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9558999999999999e-05 2023-10-20T03:35:41.175 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.175 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.175 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.176 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476514+0000", 2023-10-20T03:35:41.176 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3917000000000001e-05 2023-10-20T03:35:41.176 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.176 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.177 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.177 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476666+0000", 2023-10-20T03:35:41.177 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000152356 2023-10-20T03:35:41.177 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.177 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.177 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.178 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476687+0000", 2023-10-20T03:35:41.178 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0659e-05 2023-10-20T03:35:41.178 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.178 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.179 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.179 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.487589+0000", 2023-10-20T03:35:41.179 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010901643000000001 2023-10-20T03:35:41.179 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.179 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.180 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.180 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.487655+0000", 2023-10-20T03:35:41.180 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6395000000000004e-05 2023-10-20T03:35:41.180 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.180 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.181 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.181 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.181 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.181 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s1 54/31 ECSubWrite(tid=67, reqid=client.4570.0:420, at_version=54'34, trim_to=0'0, roll_forward_to=54'32))", 2023-10-20T03:35:41.181 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.488168+0000", 2023-10-20T03:35:41.181 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.4708658689999998, 2023-10-20T03:35:41.182 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.009928872, 2023-10-20T03:35:41.182 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.182 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.182 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.182 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.183 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.183 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488168+0000", 2023-10-20T03:35:41.183 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.183 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.183 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.184 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.184 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488168+0000", 2023-10-20T03:35:41.184 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.184 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.184 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.184 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.185 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488170+0000", 2023-10-20T03:35:41.185 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5099999999999999e-06 2023-10-20T03:35:41.185 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.185 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.185 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.185 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488279+0000", 2023-10-20T03:35:41.186 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000109547 2023-10-20T03:35:41.186 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.186 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.186 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.186 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488319+0000", 2023-10-20T03:35:41.187 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9709000000000002e-05 2023-10-20T03:35:41.187 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.187 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.187 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.187 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488342+0000", 2023-10-20T03:35:41.188 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.351e-05 2023-10-20T03:35:41.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.188 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.188 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.188 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488486+0000", 2023-10-20T03:35:41.189 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000143609 2023-10-20T03:35:41.189 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.189 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.189 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.189 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488510+0000", 2023-10-20T03:35:41.190 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3961999999999999e-05 2023-10-20T03:35:41.190 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.190 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.190 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.190 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.498021+0000", 2023-10-20T03:35:41.190 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0095113769999999997 2023-10-20T03:35:41.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.191 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.191 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.191 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.498097+0000", 2023-10-20T03:35:41.191 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5648000000000005e-05 2023-10-20T03:35:41.192 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.192 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.192 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.192 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.192 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.192 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:41.194 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-10-20T03:35:41.195 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:41.192+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:41.295 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:41.290+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:41.326 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:41.327 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:41.327 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2023-10-20T03:35:41.327 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:41.341 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-10-20T03:35:41.396 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:41.392+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:41.497 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:41.492+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:41.506 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:41.506 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:41.507 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:35:41.507 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:35:41.507 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:41.520 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-10-20T03:35:41.598 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:41.597+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:41.636 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:41.637 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2023-10-20T03:35:41.637 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2023-10-20T03:35:41.637 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2023-10-20T03:35:41.637 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.637 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:208 3.es0 3:7dffa048:::smithi17039300-5:head [append 0~688128] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:41.637 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.269565+0000", 2023-10-20T03:35:41.638 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.366202584, 2023-10-20T03:35:41.638 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029923571, 2023-10-20T03:35:41.638 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.638 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:41.638 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:41.639 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:41.639 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:41.639 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 208 2023-10-20T03:35:41.639 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.639 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.640 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.640 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.640 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.269565+0000", 2023-10-20T03:35:41.640 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.640 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.640 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.641 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.641 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.269565+0000", 2023-10-20T03:35:41.641 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.641 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.641 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.641 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.642 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.269569+0000", 2023-10-20T03:35:41.642 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0230000000000001e-06 2023-10-20T03:35:41.642 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.642 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.642 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.643 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270337+0000", 2023-10-20T03:35:41.643 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00076892900000000001 2023-10-20T03:35:41.643 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.643 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.643 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.644 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270372+0000", 2023-10-20T03:35:41.644 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4652e-05 2023-10-20T03:35:41.644 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.644 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.644 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.645 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270391+0000", 2023-10-20T03:35:41.645 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8819000000000001e-05 2023-10-20T03:35:41.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.645 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.645 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.646 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270555+0000", 2023-10-20T03:35:41.646 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016383700000000001 2023-10-20T03:35:41.646 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.646 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.646 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:41.647 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270730+0000", 2023-10-20T03:35:41.647 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017536 2023-10-20T03:35:41.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.647 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.647 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.648 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.272967+0000", 2023-10-20T03:35:41.648 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002236601 2023-10-20T03:35:41.648 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.648 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.648 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.649 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298706+0000", 2023-10-20T03:35:41.649 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025739482000000001 2023-10-20T03:35:41.649 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.649 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.649 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:41.650 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299042+0000", 2023-10-20T03:35:41.650 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00033608100000000001 2023-10-20T03:35:41.650 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.650 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.650 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.651 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299489+0000", 2023-10-20T03:35:41.651 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00044678699999999998 2023-10-20T03:35:41.651 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.651 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.651 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.652 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.652 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.652 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:209 3.es0 3:7dffa048:::smithi17039300-5:head [append 0~376832] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:41.652 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.277974+0000", 2023-10-20T03:35:41.652 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.357793748999999, 2023-10-20T03:35:41.653 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.023138072999999999, 2023-10-20T03:35:41.653 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.653 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:41.653 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:41.653 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:41.653 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:41.654 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 209 2023-10-20T03:35:41.654 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.654 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.654 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.654 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.655 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.277974+0000", 2023-10-20T03:35:41.655 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.655 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.655 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.277974+0000", 2023-10-20T03:35:41.656 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.656 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.656 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.656 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.656 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.277977+0000", 2023-10-20T03:35:41.656 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.553e-06 2023-10-20T03:35:41.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.657 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.657 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.657 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278381+0000", 2023-10-20T03:35:41.657 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00040455200000000003 2023-10-20T03:35:41.658 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.658 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.658 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.658 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278416+0000", 2023-10-20T03:35:41.658 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4269e-05 2023-10-20T03:35:41.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.659 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.659 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.659 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278440+0000", 2023-10-20T03:35:41.659 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3926000000000001e-05 2023-10-20T03:35:41.660 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.660 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.660 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.660 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.287724+0000", 2023-10-20T03:35:41.660 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0092847510000000008 2023-10-20T03:35:41.661 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.661 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.661 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:41.661 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.288029+0000", 2023-10-20T03:35:41.661 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00030455899999999998 2023-10-20T03:35:41.661 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.662 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.662 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.662 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.289828+0000", 2023-10-20T03:35:41.662 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017987210000000001 2023-10-20T03:35:41.663 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.663 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.663 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.663 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.300205+0000", 2023-10-20T03:35:41.663 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010377752000000001 2023-10-20T03:35:41.663 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.664 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.664 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:41.664 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.300337+0000", 2023-10-20T03:35:41.664 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000131755 2023-10-20T03:35:41.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.665 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.665 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.301112+0000", 2023-10-20T03:35:41.665 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00077523499999999999 2023-10-20T03:35:41.665 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.666 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.666 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.666 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:210 3.es0 3:7dffa048:::smithi17039300-5:head [setxattr _header (55) in=7b] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:41.667 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.278490+0000", 2023-10-20T03:35:41.667 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.357277871000001, 2023-10-20T03:35:41.667 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027549357999999999, 2023-10-20T03:35:41.667 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.668 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:41.668 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:41.668 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:41.668 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:41.668 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210 2023-10-20T03:35:41.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.669 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.669 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.669 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278490+0000", 2023-10-20T03:35:41.669 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.670 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278490+0000", 2023-10-20T03:35:41.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.671 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.671 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278492+0000", 2023-10-20T03:35:41.671 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.59e-06 2023-10-20T03:35:41.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.671 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.672 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278501+0000", 2023-10-20T03:35:41.672 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.986e-06 2023-10-20T03:35:41.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.672 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.673 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.673 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278519+0000", 2023-10-20T03:35:41.673 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8238999999999999e-05 2023-10-20T03:35:41.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.674 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.674 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278536+0000", 2023-10-20T03:35:41.674 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7376000000000001e-05 2023-10-20T03:35:41.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.675 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.675 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.301212+0000", 2023-10-20T03:35:41.675 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.022675563999999999 2023-10-20T03:35:41.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.675 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:41.676 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.301368+0000", 2023-10-20T03:35:41.676 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015616 2023-10-20T03:35:41.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.676 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.677 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.302117+0000", 2023-10-20T03:35:41.677 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00074924300000000002 2023-10-20T03:35:41.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.677 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.677 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.303929+0000", 2023-10-20T03:35:41.678 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0018115830000000001 2023-10-20T03:35:41.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.678 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:41.678 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.305714+0000", 2023-10-20T03:35:41.679 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001785502 2023-10-20T03:35:41.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.679 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.679 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.306040+0000", 2023-10-20T03:35:41.679 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00032511500000000001 2023-10-20T03:35:41.679 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.680 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.680 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.680 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:211 3.es0 3:7dffa048:::smithi17039300-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:41.680 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.278563+0000", 2023-10-20T03:35:41.681 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.357205564000001, 2023-10-20T03:35:41.681 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030205164, 2023-10-20T03:35:41.681 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.681 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:41.681 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:41.681 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:41.682 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:41.682 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211 2023-10-20T03:35:41.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.682 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.682 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.683 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.683 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278563+0000", 2023-10-20T03:35:41.683 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.683 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.684 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278563+0000", 2023-10-20T03:35:41.684 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.684 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.685 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278564+0000", 2023-10-20T03:35:41.685 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.826e-06 2023-10-20T03:35:41.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.685 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.685 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278572+0000", 2023-10-20T03:35:41.686 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0299999999999994e-06 2023-10-20T03:35:41.686 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.686 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.686 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278598+0000", 2023-10-20T03:35:41.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5593e-05 2023-10-20T03:35:41.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278619+0000", 2023-10-20T03:35:41.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0755e-05 2023-10-20T03:35:41.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.302759+0000", 2023-10-20T03:35:41.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.024140663 2023-10-20T03:35:41.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.689 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:41.690 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.303005+0000", 2023-10-20T03:35:41.690 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024529499999999997 2023-10-20T03:35:41.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.690 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.691 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.306691+0000", 2023-10-20T03:35:41.691 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0036863809999999999 2023-10-20T03:35:41.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.691 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:41.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.306820+0000", 2023-10-20T03:35:41.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012892200000000001 2023-10-20T03:35:41.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.693 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.308768+0000", 2023-10-20T03:35:41.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001947699 2023-10-20T03:35:41.693 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.693 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.694 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.694 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=23, last_complete=37'14, committed=1, applied=1))", 2023-10-20T03:35:41.694 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.282066+0000", 2023-10-20T03:35:41.695 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.353701937, 2023-10-20T03:35:41.695 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.01609031, 2023-10-20T03:35:41.695 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.695 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.696 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.696 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282066+0000", 2023-10-20T03:35:41.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282066+0000", 2023-10-20T03:35:41.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.699 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.699 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282074+0000", 2023-10-20T03:35:41.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9840000000000001e-06 2023-10-20T03:35:41.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.700 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282084+0000", 2023-10-20T03:35:41.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0020000000000001e-05 2023-10-20T03:35:41.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282113+0000", 2023-10-20T03:35:41.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8572e-05 2023-10-20T03:35:41.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282134+0000", 2023-10-20T03:35:41.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1622000000000001e-05 2023-10-20T03:35:41.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.704 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.704 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298102+0000", 2023-10-20T03:35:41.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015968110000000001 2023-10-20T03:35:41.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298156+0000", 2023-10-20T03:35:41.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4002000000000002e-05 2023-10-20T03:35:41.706 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.706 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.706 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.707 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.707 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=23, last_complete=37'14, committed=1, applied=1))", 2023-10-20T03:35:41.707 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.282628+0000", 2023-10-20T03:35:41.707 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.353139769999999, 2023-10-20T03:35:41.707 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015776847, 2023-10-20T03:35:41.708 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.708 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.708 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.708 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282628+0000", 2023-10-20T03:35:41.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282628+0000", 2023-10-20T03:35:41.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282630+0000", 2023-10-20T03:35:41.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9199999999999998e-06 2023-10-20T03:35:41.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282640+0000", 2023-10-20T03:35:41.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0056e-05 2023-10-20T03:35:41.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.713 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282670+0000", 2023-10-20T03:35:41.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0114999999999999e-05 2023-10-20T03:35:41.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.282694+0000", 2023-10-20T03:35:41.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3623000000000001e-05 2023-10-20T03:35:41.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.715 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.715 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298309+0000", 2023-10-20T03:35:41.715 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015614663000000001 2023-10-20T03:35:41.715 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.715 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.716 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.716 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298405+0000", 2023-10-20T03:35:41.716 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6470000000000003e-05 2023-10-20T03:35:41.716 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.716 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.717 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.717 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=23, last_complete=37'14, committed=1, applied=1))", 2023-10-20T03:35:41.718 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.283349+0000", 2023-10-20T03:35:41.718 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.352419313999999, 2023-10-20T03:35:41.718 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015191886, 2023-10-20T03:35:41.718 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.718 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.719 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.719 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.719 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.283349+0000", 2023-10-20T03:35:41.719 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.283349+0000", 2023-10-20T03:35:41.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.721 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.721 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.283351+0000", 2023-10-20T03:35:41.721 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2369999999999999e-06 2023-10-20T03:35:41.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.283364+0000", 2023-10-20T03:35:41.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2605999999999999e-05 2023-10-20T03:35:41.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.723 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.723 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.283401+0000", 2023-10-20T03:35:41.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7154999999999999e-05 2023-10-20T03:35:41.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.724 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.724 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.283422+0000", 2023-10-20T03:35:41.724 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0936000000000001e-05 2023-10-20T03:35:41.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298487+0000", 2023-10-20T03:35:41.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015064889999999999 2023-10-20T03:35:41.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.726 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.726 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298541+0000", 2023-10-20T03:35:41.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4061999999999998e-05 2023-10-20T03:35:41.726 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.726 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.726 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.727 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.ds2 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:41.727 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.388826+0000", 2023-10-20T03:35:41.727 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 15.246942475000001, 2023-10-20T03:35:41.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027218437000000002, 2023-10-20T03:35:41.728 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.728 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.728 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388826+0000", 2023-10-20T03:35:41.729 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.730 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388826+0000", 2023-10-20T03:35:41.730 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.730 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388827+0000", 2023-10-20T03:35:41.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5300000000000002e-07 2023-10-20T03:35:41.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389257+0000", 2023-10-20T03:35:41.732 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00043021500000000001 2023-10-20T03:35:41.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.732 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.732 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389281+0000", 2023-10-20T03:35:41.733 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4697000000000001e-05 2023-10-20T03:35:41.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.733 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.733 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.733 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389295+0000", 2023-10-20T03:35:41.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3491e-05 2023-10-20T03:35:41.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.734 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.734 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389419+0000", 2023-10-20T03:35:41.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012411400000000001 2023-10-20T03:35:41.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.735 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.735 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389432+0000", 2023-10-20T03:35:41.735 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2551000000000001e-05 2023-10-20T03:35:41.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.736 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.736 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.736 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.415986+0000", 2023-10-20T03:35:41.736 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026554069999999999 2023-10-20T03:35:41.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.737 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.737 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.737 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.416044+0000", 2023-10-20T03:35:41.737 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8346e-05 2023-10-20T03:35:41.738 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.738 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.738 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.738 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.738 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.739 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.ds2 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:41.739 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.395618+0000", 2023-10-20T03:35:41.739 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 15.240150338999999, 2023-10-20T03:35:41.739 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.02052816, 2023-10-20T03:35:41.739 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.740 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.740 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.740 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.740 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.740 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395618+0000", 2023-10-20T03:35:41.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.741 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.741 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.741 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395618+0000", 2023-10-20T03:35:41.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.742 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395619+0000", 2023-10-20T03:35:41.742 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5400000000000001e-06 2023-10-20T03:35:41.743 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.743 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.743 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.743 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395915+0000", 2023-10-20T03:35:41.743 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00029547800000000001 2023-10-20T03:35:41.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.744 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.744 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.744 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395953+0000", 2023-10-20T03:35:41.744 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8164000000000003e-05 2023-10-20T03:35:41.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.745 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.745 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395971+0000", 2023-10-20T03:35:41.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8484000000000001e-05 2023-10-20T03:35:41.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.746 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.746 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.399211+0000", 2023-10-20T03:35:41.746 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0032393040000000001 2023-10-20T03:35:41.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.747 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.747 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.747 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.399239+0000", 2023-10-20T03:35:41.747 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7991999999999999e-05 2023-10-20T03:35:41.748 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.748 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.748 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.748 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.416107+0000", 2023-10-20T03:35:41.748 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016868095 2023-10-20T03:35:41.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.749 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.749 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.749 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.416146+0000", 2023-10-20T03:35:41.749 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9103000000000002e-05 2023-10-20T03:35:41.750 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.750 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.750 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.750 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.750 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.751 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.5s6 44/31 ECSubWrite(tid=22, reqid=client.4570.0:340, at_version=44'12, trim_to=0'0, roll_forward_to=43'10))", 2023-10-20T03:35:41.751 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.399843+0000", 2023-10-20T03:35:41.751 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 15.235924666000001, 2023-10-20T03:35:41.751 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016250701999999999, 2023-10-20T03:35:41.751 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.751 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.752 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.752 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.752 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.399843+0000", 2023-10-20T03:35:41.752 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.753 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.753 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.753 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.753 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.399843+0000", 2023-10-20T03:35:41.753 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.754 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.754 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.754 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.754 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.399844+0000", 2023-10-20T03:35:41.754 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4300000000000002e-07 2023-10-20T03:35:41.755 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.755 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.755 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.755 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.400098+0000", 2023-10-20T03:35:41.755 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00025422500000000001 2023-10-20T03:35:41.756 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.756 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.756 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.756 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.400140+0000", 2023-10-20T03:35:41.756 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1412000000000003e-05 2023-10-20T03:35:41.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.757 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.757 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.757 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.400164+0000", 2023-10-20T03:35:41.757 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3779999999999999e-05 2023-10-20T03:35:41.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.758 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.758 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.400436+0000", 2023-10-20T03:35:41.758 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000272355 2023-10-20T03:35:41.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.759 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.759 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.759 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.400455+0000", 2023-10-20T03:35:41.759 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8813000000000001e-05 2023-10-20T03:35:41.759 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.760 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.760 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.760 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.416054+0000", 2023-10-20T03:35:41.760 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015598888 2023-10-20T03:35:41.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.761 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.761 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.761 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.416094+0000", 2023-10-20T03:35:41.761 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0485999999999998e-05 2023-10-20T03:35:41.761 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.762 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.762 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.762 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.762 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.ds2 44/31 ECSubWrite(tid=53, reqid=client.4570.0:335, at_version=44'20, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:41.763 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.403512+0000", 2023-10-20T03:35:41.763 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 15.232255917, 2023-10-20T03:35:41.763 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012714655, 2023-10-20T03:35:41.763 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.763 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.764 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.764 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.764 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.764 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.403512+0000", 2023-10-20T03:35:41.764 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.765 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.765 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.765 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.403512+0000", 2023-10-20T03:35:41.765 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.765 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.766 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.766 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.766 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.403513+0000", 2023-10-20T03:35:41.766 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3e-06 2023-10-20T03:35:41.766 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.767 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.767 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.767 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.403564+0000", 2023-10-20T03:35:41.767 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0046999999999997e-05 2023-10-20T03:35:41.767 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.768 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.768 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.768 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.403593+0000", 2023-10-20T03:35:41.768 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9043e-05 2023-10-20T03:35:41.768 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.769 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.769 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.769 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.403608+0000", 2023-10-20T03:35:41.769 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5505000000000001e-05 2023-10-20T03:35:41.769 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.769 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.770 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.770 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.413890+0000", 2023-10-20T03:35:41.770 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010282316 2023-10-20T03:35:41.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.770 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.771 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.771 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.413908+0000", 2023-10-20T03:35:41.771 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8114e-05 2023-10-20T03:35:41.771 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.771 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.771 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.772 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.416173+0000", 2023-10-20T03:35:41.772 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0022644259999999999 2023-10-20T03:35:41.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.772 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.772 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.773 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.416227+0000", 2023-10-20T03:35:41.773 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3903999999999998e-05 2023-10-20T03:35:41.773 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.773 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.773 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.774 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.774 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.774 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.ds2 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:35:41.774 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:30.392840+0000", 2023-10-20T03:35:41.774 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 11.242928269, 2023-10-20T03:35:41.775 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.037253397000000001, 2023-10-20T03:35:41.775 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.775 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.775 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.775 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.775 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.776 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.392840+0000", 2023-10-20T03:35:41.776 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.776 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.777 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.777 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.392840+0000", 2023-10-20T03:35:41.777 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.777 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.778 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.392841+0000", 2023-10-20T03:35:41.778 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.567e-06 2023-10-20T03:35:41.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.778 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.778 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.779 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.394858+0000", 2023-10-20T03:35:41.779 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020165589999999998 2023-10-20T03:35:41.779 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.779 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.779 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.780 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.394892+0000", 2023-10-20T03:35:41.780 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4001999999999997e-05 2023-10-20T03:35:41.780 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.780 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.780 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.781 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.394916+0000", 2023-10-20T03:35:41.781 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4002000000000002e-05 2023-10-20T03:35:41.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.781 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.781 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.782 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.395085+0000", 2023-10-20T03:35:41.782 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016879899999999999 2023-10-20T03:35:41.782 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.782 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.782 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.783 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.395112+0000", 2023-10-20T03:35:41.783 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7032000000000001e-05 2023-10-20T03:35:41.783 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.783 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.783 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.784 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.430024+0000", 2023-10-20T03:35:41.784 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.034911958 2023-10-20T03:35:41.784 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.784 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.784 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.784 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.430093+0000", 2023-10-20T03:35:41.785 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9478000000000001e-05 2023-10-20T03:35:41.785 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.785 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.785 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.785 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.785 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.786 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs5 53/47 ECSubWrite(tid=56, reqid=osd.3.0:55, at_version=0'0, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:41.786 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.452318+0000", 2023-10-20T03:35:41.786 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 6.1834496679999997, 2023-10-20T03:35:41.786 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019749735000000001, 2023-10-20T03:35:41.786 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.787 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.787 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.787 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.787 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.787 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452318+0000", 2023-10-20T03:35:41.788 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.788 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.788 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.788 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.788 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452318+0000", 2023-10-20T03:35:41.788 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.789 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.789 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.789 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.789 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452321+0000", 2023-10-20T03:35:41.789 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2670000000000001e-06 2023-10-20T03:35:41.790 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.790 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.790 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.790 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452350+0000", 2023-10-20T03:35:41.790 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8915999999999999e-05 2023-10-20T03:35:41.790 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.791 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.791 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.791 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452391+0000", 2023-10-20T03:35:41.791 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0992000000000003e-05 2023-10-20T03:35:41.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.792 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.792 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.792 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452414+0000", 2023-10-20T03:35:41.792 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3632999999999999e-05 2023-10-20T03:35:41.792 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.792 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.793 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.793 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452600+0000", 2023-10-20T03:35:41.793 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018559199999999999 2023-10-20T03:35:41.793 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.793 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.793 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.794 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452632+0000", 2023-10-20T03:35:41.794 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2298999999999997e-05 2023-10-20T03:35:41.794 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.794 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.794 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.795 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.471913+0000", 2023-10-20T03:35:41.795 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019280950000000002 2023-10-20T03:35:41.795 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.795 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.795 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.795 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.472068+0000", 2023-10-20T03:35:41.796 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000155086 2023-10-20T03:35:41.796 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.796 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.796 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.796 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.796 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.797 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs5 53/47 ECSubWrite(tid=57, reqid=client.4570.0:408, at_version=53'31, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:41.797 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.452469+0000", 2023-10-20T03:35:41.797 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 6.1832987179999996, 2023-10-20T03:35:41.797 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.033778967999999999, 2023-10-20T03:35:41.797 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.797 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.798 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.798 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.798 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.798 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452469+0000", 2023-10-20T03:35:41.798 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.799 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.799 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.799 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.799 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452469+0000", 2023-10-20T03:35:41.799 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.800 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.800 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.800 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.800 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452470+0000", 2023-10-20T03:35:41.800 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6299999999999993e-07 2023-10-20T03:35:41.801 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.801 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.801 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.801 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452906+0000", 2023-10-20T03:35:41.801 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00043586900000000002 2023-10-20T03:35:41.801 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.802 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.802 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.802 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452933+0000", 2023-10-20T03:35:41.802 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.69e-05 2023-10-20T03:35:41.802 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.803 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.803 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.803 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452952+0000", 2023-10-20T03:35:41.803 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9089e-05 2023-10-20T03:35:41.803 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.804 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.804 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.804 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.453188+0000", 2023-10-20T03:35:41.804 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00023577799999999999 2023-10-20T03:35:41.804 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.805 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.805 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.805 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.453218+0000", 2023-10-20T03:35:41.805 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0442000000000001e-05 2023-10-20T03:35:41.805 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.806 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.806 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.806 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.486138+0000", 2023-10-20T03:35:41.806 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.032919118999999997 2023-10-20T03:35:41.806 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.807 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.807 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.807 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.486248+0000", 2023-10-20T03:35:41.807 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110808 2023-10-20T03:35:41.807 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.808 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.808 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.808 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.808 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.809 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs5 53/47 ECSubWrite(tid=58, reqid=client.4570.0:409, at_version=53'32, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:41.809 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.473108+0000", 2023-10-20T03:35:41.809 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 6.1626599410000003, 2023-10-20T03:35:41.809 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015486623, 2023-10-20T03:35:41.809 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.810 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.810 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.810 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.810 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.810 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473108+0000", 2023-10-20T03:35:41.811 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.811 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.811 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.811 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.812 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473108+0000", 2023-10-20T03:35:41.812 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.812 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.812 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.812 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.813 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473110+0000", 2023-10-20T03:35:41.813 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5960000000000001e-06 2023-10-20T03:35:41.813 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.813 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.813 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.813 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473468+0000", 2023-10-20T03:35:41.814 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000358518 2023-10-20T03:35:41.814 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.814 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.814 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.814 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473499+0000", 2023-10-20T03:35:41.814 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0882e-05 2023-10-20T03:35:41.815 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.815 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.815 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.815 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473522+0000", 2023-10-20T03:35:41.815 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2623e-05 2023-10-20T03:35:41.816 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.816 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.816 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.816 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473704+0000", 2023-10-20T03:35:41.816 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000182616 2023-10-20T03:35:41.816 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.817 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.817 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.817 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473773+0000", 2023-10-20T03:35:41.817 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9093999999999999e-05 2023-10-20T03:35:41.817 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.818 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.818 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.818 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.488509+0000", 2023-10-20T03:35:41.818 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014735506000000001 2023-10-20T03:35:41.818 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.819 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.819 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.819 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.488595+0000", 2023-10-20T03:35:41.819 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5787999999999999e-05 2023-10-20T03:35:41.819 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.820 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.820 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.820 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.820 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.820 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:414 3.es0 3:7dffa048:::smithi17039300-5:head [create,setxattr gochgegikXngWahUkkjRl^QTgYhb[b]WXamd_eSq (2384) in=40b,setxattr b\\PtoYR`hmWtnd_a[S\\\\lXhWq]eThtkqjphngsWR (1473) in=40b,setxattr Ub[Vfm^UiUZrRqc[oNaY`rgk\\rgTkNpothWnk[oU (771) in=40b,setxattr OOq^bpU^bm (1437) in=10b,setxattr biYdnofqNd_eZg\\amb``Na_saZ^ (1437) in=27b,setxattr rX`YVl[kmaRi[QfikjQasTjWZSRiWrW`bkcandUf (35) in=40b,setxattr [YUqn^[Zb^O\\a_bmaVfNcegQesrbqaZSZoip[Xdr (5752) in=40b,setxattr fceY\\cr[bnaj[ (1437) in=13b,setxattr acNNOX`hScRmkbQpOdbgSnbYo_aTYX^]SiYpcXhV (1435) in=40b,setxattr olt]ab (1437) in=6b,setxattr UlYSXgbei[Yp^prXjdVWknUpaYc`bdacfOTgrmOe (279) in=40b,setxattr Pe[lpnRflfpehZ[\\\\niYcpg]NTejdgUaQ\\^PY[pQ (1122) in=40b,setxattr \\oimfXol\\dqgos_Rkh\\U`blYV\\`qdZco_d`fnl\\a (877) in=40b,setxattr ahhrfeWh_qaa_ohbtahY_RbYoQU`Y`R[bTOOhsgS (593) in=40b,setxattr iUbWaVpfdgVcRsPQUtUOtQdYdVWantYNjbgOPU^t (842) in=40b,setxattr \\R^[`cSkXTlebhQo^[nNOpilmsejaimZd[tgWYb_ (1438) in=40b,setxattr PbnTj^iPVhr`VaY`i[oQ[Pj]b[ZjaaSgYfVfWhrX (2805) in=40b,setxattr NSphstUlmcX]Uhs]jPZ^nXY]Rh^oVkSZfn`[OQRZ (1542) in=40b,setxattr Ysnsrf[rY`mflrhPbnkV_\\dRrQPeld[oXqV[Stcn (842) in=40b,setxattr \\rht]cXhNbU__TUnbohctXit^XTsnoTUNTPNY[fp (1962) in=40b] snapc f=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-10-20T03:35:41.821 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.427647+0000", 2023-10-20T03:35:41.821 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 5.2081205940000004, 2023-10-20T03:35:41.821 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.018072287999999999, 2023-10-20T03:35:41.821 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.822 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:41.822 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:41.822 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:41.822 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:41.822 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 414 2023-10-20T03:35:41.822 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.823 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.823 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.823 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.823 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.427647+0000", 2023-10-20T03:35:41.823 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.824 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.824 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.824 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.824 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.427647+0000", 2023-10-20T03:35:41.824 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.825 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.825 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.825 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.825 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.427650+0000", 2023-10-20T03:35:41.825 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.277e-06 2023-10-20T03:35:41.826 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.826 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.826 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.826 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.427694+0000", 2023-10-20T03:35:41.826 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3865000000000001e-05 2023-10-20T03:35:41.826 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.827 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.827 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.827 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.427724+0000", 2023-10-20T03:35:41.827 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0043e-05 2023-10-20T03:35:41.827 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.828 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.828 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.828 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.427741+0000", 2023-10-20T03:35:41.828 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7266e-05 2023-10-20T03:35:41.828 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.828 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.829 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.829 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.427889+0000", 2023-10-20T03:35:41.829 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014830900000000001 2023-10-20T03:35:41.829 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.829 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.830 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:41.830 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.428246+0000", 2023-10-20T03:35:41.830 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00035656799999999998 2023-10-20T03:35:41.830 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.830 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.831 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.831 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.430374+0000", 2023-10-20T03:35:41.831 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0021277449999999999 2023-10-20T03:35:41.831 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.831 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.832 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.832 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.433523+0000", 2023-10-20T03:35:41.832 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0031495690000000001 2023-10-20T03:35:41.832 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.832 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.832 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:41.833 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.445203+0000", 2023-10-20T03:35:41.833 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011679968000000001 2023-10-20T03:35:41.833 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.833 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.833 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.834 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.445720+0000", 2023-10-20T03:35:41.834 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00051667800000000002 2023-10-20T03:35:41.834 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.834 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.834 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.835 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.835 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.835 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:432 3.es0 3:7b3c804c:::smithi17039300-40:head [create,setxattr [VYQXaeer[Nsq`eTbgjNZfdhqbTRn_][lloSeZUP (16840) in=40b,setxattr hpaXc^]qjrqU_ciY[cWncm]SPa^tP_jRrRXm`jol (175) in=40b,setxattr YVpShpT[fhfamkUsgr]mb\\foasmX^k`jdhbbdlfP (1823) in=40b,setxattr ZhfkQl`OllQo^hemUnadbmqWdoabfgjcc^sVm_gf (1611) in=40b,setxattr aakOOaWemYRPYcaZ\\knQb (1051) in=21b,setxattr gPaagTUadWkOtqaRt]^rdUj^Zinmt]U^ (1051) in=32b,setxattr SfXmSZ]crX[dqcaqjNrrroQQbptW^Q^nUSUlcgdk (807) in=40b,setxattr SmRaehVPpUNpfZ[UX\\ZO_Ygfofk^ah^Xki^XTeX` (1191) in=40b,setxattr aTgW^YtXrjnrhT (1051) in=14b,setxattr idO[Vrlsc^pW[hYfh`UQSs^m_mU[YefWeld[YZ[c (1543) in=40b,setxattr ]Q``task\\rr]qaY[Pg`i[alh[rPlSdrcQ^iW^Z^g (1157) in=40b,setxattr _OOWpoQQg\\jtQUi``q_s[ae_akq`YOYilNTq^SPO (1121) in=40b,setxattr kN\\_ZaOX]q[kXen^Z]liYfaVP]prk\\r\\\\_Wsd[d_ (105) in=40b,setxattr OU`jdr^jTir[joOegmm (1051) in=19b,setxattr tRofbjROgtWqoRfmUVZXhpaZ[Q^sUNZb`\\YsdUfh (1507) in=40b,setxattr RRNpbt[`okmbR]RpSWfmmedXrUpaltcbWPbNRZRZ (67) in=40b,setxattr i[`^r^aojXRaU[[T\\kgWN^ (1051) in=22b,setxattr gZhSajUgdm`o]W[X`sgdVl^TtY^_SX[_l[Tr_aaO (663) in=40b,setxattr qN][QajljeddtQTcQX_QaTrg\\V\\pY\\QbsWehde`s (1646) in=40b,setxattr fbTrNt\\`ZVPVN[]tcg]iPXYOnmYnV]oaW_^o^gdr (1403) in=40b] snapc 16=[16,3,2] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2023-10-20T03:35:41.835 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:37.439657+0000", 2023-10-20T03:35:41.835 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.1961114989999997, 2023-10-20T03:35:41.836 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.021990928, 2023-10-20T03:35:41.836 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.836 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:41.836 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:41.836 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:41.837 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:41.837 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 432 2023-10-20T03:35:41.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.837 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.837 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.838 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.838 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439657+0000", 2023-10-20T03:35:41.838 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.838 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.838 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.838 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.839 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439657+0000", 2023-10-20T03:35:41.839 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.839 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.839 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.840 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439660+0000", 2023-10-20T03:35:41.840 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.433e-06 2023-10-20T03:35:41.840 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.840 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.840 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.841 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439750+0000", 2023-10-20T03:35:41.841 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9494999999999996e-05 2023-10-20T03:35:41.841 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.841 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.841 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.842 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439790+0000", 2023-10-20T03:35:41.842 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0914999999999997e-05 2023-10-20T03:35:41.842 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.842 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.842 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.843 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439815+0000", 2023-10-20T03:35:41.843 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4586000000000001e-05 2023-10-20T03:35:41.843 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.843 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.843 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.844 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439997+0000", 2023-10-20T03:35:41.844 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000181893 2023-10-20T03:35:41.844 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.844 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.844 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op must wait for map", 2023-10-20T03:35:41.845 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.440020+0000", 2023-10-20T03:35:41.845 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2685999999999999e-05 2023-10-20T03:35:41.845 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.845 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.845 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.845 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.445780+0000", 2023-10-20T03:35:41.846 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.005760818 2023-10-20T03:35:41.846 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.846 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.846 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:41.846 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.445992+0000", 2023-10-20T03:35:41.847 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00021181699999999999 2023-10-20T03:35:41.847 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.847 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.847 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.847 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.447439+0000", 2023-10-20T03:35:41.848 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0014467530000000001 2023-10-20T03:35:41.848 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.848 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.848 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.848 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.451095+0000", 2023-10-20T03:35:41.849 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0036556359999999999 2023-10-20T03:35:41.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.849 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.849 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:41.849 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.461212+0000", 2023-10-20T03:35:41.850 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010117477 2023-10-20T03:35:41.850 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.850 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.850 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.850 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.461648+0000", 2023-10-20T03:35:41.851 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00043541899999999999 2023-10-20T03:35:41.851 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.851 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.851 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.851 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.852 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.852 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.3s4 55/54 ECSubWrite(tid=30, reqid=client.4570.0:423, at_version=55'21, trim_to=0'0, roll_forward_to=0'0))", 2023-10-20T03:35:41.852 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:37.466402+0000", 2023-10-20T03:35:41.852 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.1693661799999999, 2023-10-20T03:35:41.853 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013402259, 2023-10-20T03:35:41.853 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.853 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.853 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.853 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.854 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.854 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.466402+0000", 2023-10-20T03:35:41.854 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.854 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.854 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.855 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.855 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.466402+0000", 2023-10-20T03:35:41.855 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.855 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.855 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.856 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.856 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.466403+0000", 2023-10-20T03:35:41.856 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3600000000000002e-07 2023-10-20T03:35:41.856 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.856 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.857 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.857 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.467185+0000", 2023-10-20T03:35:41.857 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00078215200000000004 2023-10-20T03:35:41.857 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.857 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.858 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.858 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.467216+0000", 2023-10-20T03:35:41.858 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0788999999999999e-05 2023-10-20T03:35:41.858 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.858 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.859 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.859 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.467232+0000", 2023-10-20T03:35:41.859 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6362999999999999e-05 2023-10-20T03:35:41.859 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.859 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.860 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.860 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.467316+0000", 2023-10-20T03:35:41.860 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3489e-05 2023-10-20T03:35:41.860 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.860 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.861 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.861 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.467326+0000", 2023-10-20T03:35:41.861 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.025e-05 2023-10-20T03:35:41.861 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.861 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.862 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.862 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.479694+0000", 2023-10-20T03:35:41.862 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012367912 2023-10-20T03:35:41.862 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.862 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.863 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.863 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.479804+0000", 2023-10-20T03:35:41.863 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110468 2023-10-20T03:35:41.863 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.863 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.864 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.864 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.864 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.864 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs5 55/47 ECSubWrite(tid=71, reqid=client.4570.0:406, at_version=55'38, trim_to=0'0, roll_forward_to=54'36))", 2023-10-20T03:35:41.864 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:37.472798+0000", 2023-10-20T03:35:41.864 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.1629699489999998, 2023-10-20T03:35:41.865 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019968949999999999, 2023-10-20T03:35:41.865 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.865 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.865 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.865 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.865 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.866 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.472798+0000", 2023-10-20T03:35:41.866 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.866 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.866 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.866 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.867 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.472798+0000", 2023-10-20T03:35:41.867 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.867 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.867 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.867 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.868 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.472800+0000", 2023-10-20T03:35:41.868 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.654e-06 2023-10-20T03:35:41.868 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.868 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.868 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.868 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473206+0000", 2023-10-20T03:35:41.869 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00040649700000000002 2023-10-20T03:35:41.869 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.869 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.869 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.869 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473249+0000", 2023-10-20T03:35:41.869 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2691999999999997e-05 2023-10-20T03:35:41.870 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.870 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.870 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.870 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473268+0000", 2023-10-20T03:35:41.870 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9355999999999999e-05 2023-10-20T03:35:41.871 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.871 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.871 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.871 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473402+0000", 2023-10-20T03:35:41.871 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013405700000000001 2023-10-20T03:35:41.872 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.872 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.872 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.872 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473418+0000", 2023-10-20T03:35:41.872 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6028999999999998e-05 2023-10-20T03:35:41.872 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.873 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.873 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.873 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.492673+0000", 2023-10-20T03:35:41.873 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019254454000000001 2023-10-20T03:35:41.873 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.874 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.874 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.874 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.492767+0000", 2023-10-20T03:35:41.874 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4210999999999996e-05 2023-10-20T03:35:41.874 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.875 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.875 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.875 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.875 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.875 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.3s4 55/54 ECSubWrite(tid=31, reqid=client.4570.0:424, at_version=55'22, trim_to=0'0, roll_forward_to=0'0))", 2023-10-20T03:35:41.876 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:37.490125+0000", 2023-10-20T03:35:41.876 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.1456426359999998, 2023-10-20T03:35:41.876 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.01575302, 2023-10-20T03:35:41.876 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:41.876 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:41.877 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:41.877 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.877 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:41.877 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.490125+0000", 2023-10-20T03:35:41.877 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.877 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.878 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.878 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:41.878 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.490125+0000", 2023-10-20T03:35:41.878 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:41.878 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.879 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.879 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:41.879 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.490127+0000", 2023-10-20T03:35:41.879 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.666e-06 2023-10-20T03:35:41.879 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.879 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.880 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:41.880 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.490443+0000", 2023-10-20T03:35:41.880 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00031631999999999998 2023-10-20T03:35:41.880 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.880 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.881 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:41.881 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.490485+0000", 2023-10-20T03:35:41.881 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1585000000000002e-05 2023-10-20T03:35:41.881 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.881 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.882 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:41.882 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.490517+0000", 2023-10-20T03:35:41.882 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.1763000000000001e-05 2023-10-20T03:35:41.882 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.882 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.883 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:41.883 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.490712+0000", 2023-10-20T03:35:41.883 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019527100000000001 2023-10-20T03:35:41.883 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.883 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.883 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:41.884 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.490739+0000", 2023-10-20T03:35:41.884 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7308e-05 2023-10-20T03:35:41.884 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.884 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.884 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:41.885 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.505773+0000", 2023-10-20T03:35:41.885 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015033476 2023-10-20T03:35:41.885 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:41.885 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:41.885 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:41.886 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.505878+0000", 2023-10-20T03:35:41.886 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010563100000000001 2023-10-20T03:35:41.886 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.886 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.886 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.887 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:41.887 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:41.887 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:41.889 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2023-10-20T03:35:41.890 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-10-20T03:35:41.891 INFO:tasks.rados.rados.0.smithi170.stdout:191: done (0 left) 2023-10-20T03:35:41.891 INFO:tasks.rados.rados.0.smithi170.stdout:192: read oid 23 snap -1 2023-10-20T03:35:41.892 INFO:tasks.rados.rados.0.smithi170.stdout:192: expect deleted 2023-10-20T03:35:41.892 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 28 current snap is 13 2023-10-20T03:35:41.892 INFO:tasks.rados.rados.0.smithi170.stdout:193: seq_num 94 ranges {458752=425984,884736=786432,1671168=376832} 2023-10-20T03:35:41.892 INFO:tasks.rados.rados.0.smithi170.stdout:193: writing smithi17039300-28 from 458752 to 884736 tid 1 2023-10-20T03:35:41.892 INFO:tasks.rados.rados.0.smithi170.stdout:193: writing smithi17039300-28 from 884736 to 1671168 tid 2 2023-10-20T03:35:41.893 INFO:tasks.rados.rados.0.smithi170.stdout:193: writing smithi17039300-28 from 1671168 to 2048000 tid 3 2023-10-20T03:35:41.893 INFO:tasks.rados.rados.0.smithi170.stdout:194: delete oid 30 current snap is 13 2023-10-20T03:35:41.893 INFO:tasks.rados.rados.0.smithi170.stdout:192: done (2 left) 2023-10-20T03:35:41.893 INFO:tasks.rados.rados.0.smithi170.stdout:194: done (1 left) 2023-10-20T03:35:41.893 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 38 current snap is 13 2023-10-20T03:35:41.894 INFO:tasks.rados.rados.0.smithi170.stdout:195: seq_num 96 ranges {0=458752,458752=688128,1146880=475136} 2023-10-20T03:35:41.894 INFO:tasks.rados.rados.0.smithi170.stdout:195: writing smithi17039300-38 from 0 to 458752 tid 1 2023-10-20T03:35:41.894 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:41.886+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:41.895 INFO:tasks.rados.rados.0.smithi170.stdout:195: writing smithi17039300-38 from 458752 to 1146880 tid 2 2023-10-20T03:35:41.899 INFO:tasks.rados.rados.0.smithi170.stdout:195: writing smithi17039300-38 from 1146880 to 1622016 tid 3 2023-10-20T03:35:41.899 INFO:tasks.rados.rados.0.smithi170.stdout:196: read oid 40 snap -1 2023-10-20T03:35:41.900 INFO:tasks.rados.rados.0.smithi170.stdout:196: expect (ObjNum 39 snap 0 seq_num 39) 2023-10-20T03:35:41.900 INFO:tasks.rados.rados.0.smithi170.stdout:197: setattr oid 22 current snap is 13 2023-10-20T03:35:41.906 INFO:tasks.rados.rados.0.smithi170.stdout:198: read oid 1 snap -1 2023-10-20T03:35:41.907 INFO:tasks.rados.rados.0.smithi170.stdout:198: expect (ObjNum 77 snap 5 seq_num 77) 2023-10-20T03:35:41.907 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 11 current snap is 13 2023-10-20T03:35:41.907 INFO:tasks.rados.rados.0.smithi170.stdout:199: seq_num 97 ranges {1359872=475136,1835008=442368,2277376=442368,2719744=278528} 2023-10-20T03:35:41.919 INFO:tasks.rados.rados.0.smithi170.stdout:199: writing smithi17039300-11 from 1359872 to 1835008 tid 1 2023-10-20T03:35:41.922 INFO:tasks.rados.rados.0.smithi170.stdout:199: writing smithi17039300-11 from 1835008 to 2277376 tid 2 2023-10-20T03:35:41.924 INFO:tasks.rados.rados.0.smithi170.stdout:199: writing smithi17039300-11 from 2277376 to 2719744 tid 3 2023-10-20T03:35:41.926 INFO:tasks.rados.rados.0.smithi170.stdout:199: writing smithi17039300-11 from 2719744 to 2998272 tid 4 2023-10-20T03:35:41.926 INFO:tasks.rados.rados.0.smithi170.stdout:200: snap_remove snap 12 2023-10-20T03:35:41.989 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:41.984+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:42.028 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:42.029 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:42.029 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2023-10-20T03:35:42.029 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:42.043 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-10-20T03:35:42.089 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:42.084+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:42.124 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:35:42.190 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:42.189+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:42.209 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:42.209 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:42.209 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:35:42.210 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:35:42.210 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:42.224 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-10-20T03:35:42.291 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:42.290+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:42.352 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2023-10-20T03:35:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2023-10-20T03:35:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2023-10-20T03:35:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:157 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'111, mlcod=24'105)", 2023-10-20T03:35:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.051811+0000", 2023-10-20T03:35:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 39.298104903000002, 2023-10-20T03:35:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0082276899999999993, 2023-10-20T03:35:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051811+0000", 2023-10-20T03:35:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051811+0000", 2023-10-20T03:35:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051813+0000", 2023-10-20T03:35:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0470000000000001e-06 2023-10-20T03:35:42.357 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.357 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.357 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.357 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051821+0000", 2023-10-20T03:35:42.357 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1759999999999993e-06 2023-10-20T03:35:42.358 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.358 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.358 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.358 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051834+0000", 2023-10-20T03:35:42.358 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3219e-05 2023-10-20T03:35:42.358 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.359 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.359 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.359 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.051844+0000", 2023-10-20T03:35:42.359 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4569999999999994e-06 2023-10-20T03:35:42.359 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.360 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.360 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.360 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053007+0000", 2023-10-20T03:35:42.360 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00116373 2023-10-20T03:35:42.360 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.360 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.361 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:42.361 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053036+0000", 2023-10-20T03:35:42.361 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8615e-05 2023-10-20T03:35:42.361 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.361 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.362 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:42.362 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.059974+0000", 2023-10-20T03:35:42.362 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0069383539999999999 2023-10-20T03:35:42.362 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.362 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.363 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.363 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060038+0000", 2023-10-20T03:35:42.363 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4091999999999996e-05 2023-10-20T03:35:42.363 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.363 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.363 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.364 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.364 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.364 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.4212.0:158 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'112, mlcod=24'105)", 2023-10-20T03:35:42.364 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:03.052405+0000", 2023-10-20T03:35:42.364 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 39.297510402999997, 2023-10-20T03:35:42.365 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0076727749999999997, 2023-10-20T03:35:42.365 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.365 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:42.365 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.365 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.365 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.366 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052405+0000", 2023-10-20T03:35:42.366 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.366 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.366 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.366 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.367 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052405+0000", 2023-10-20T03:35:42.367 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.367 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.367 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.367 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.367 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052406+0000", 2023-10-20T03:35:42.368 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.237e-06 2023-10-20T03:35:42.368 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.368 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.368 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.368 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052462+0000", 2023-10-20T03:35:42.369 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5203e-05 2023-10-20T03:35:42.369 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.369 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.369 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.369 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052477+0000", 2023-10-20T03:35:42.370 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5559000000000001e-05 2023-10-20T03:35:42.370 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.370 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.370 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.370 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.052488+0000", 2023-10-20T03:35:42.370 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0949e-05 2023-10-20T03:35:42.371 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.371 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.371 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.371 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053461+0000", 2023-10-20T03:35:42.371 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00097277500000000005 2023-10-20T03:35:42.372 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.372 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.372 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:42.372 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.053488+0000", 2023-10-20T03:35:42.372 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7322e-05 2023-10-20T03:35:42.373 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.373 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.373 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:42.373 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060041+0000", 2023-10-20T03:35:42.373 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0065531290000000004 2023-10-20T03:35:42.374 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.374 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.374 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.374 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:03.060078+0000", 2023-10-20T03:35:42.374 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6600999999999997e-05 2023-10-20T03:35:42.374 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.375 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.375 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.375 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.375 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.375 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.6s6 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:35:42.375 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:25.357590+0000", 2023-10-20T03:35:42.376 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 16.992325914999999, 2023-10-20T03:35:42.376 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015294129, 2023-10-20T03:35:42.376 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.376 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.376 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.377 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.377 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.377 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.357590+0000", 2023-10-20T03:35:42.377 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.377 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.377 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.378 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.378 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.357590+0000", 2023-10-20T03:35:42.378 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.378 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.378 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.379 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.379 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.357590+0000", 2023-10-20T03:35:42.379 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5400000000000002e-07 2023-10-20T03:35:42.379 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.379 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.379 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.379 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.358304+0000", 2023-10-20T03:35:42.380 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00071352399999999999 2023-10-20T03:35:42.380 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.380 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.380 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.380 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.358341+0000", 2023-10-20T03:35:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7917999999999998e-05 2023-10-20T03:35:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.358367+0000", 2023-10-20T03:35:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5043000000000001e-05 2023-10-20T03:35:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.358584+0000", 2023-10-20T03:35:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000217915 2023-10-20T03:35:42.383 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.383 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.383 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.383 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.358610+0000", 2023-10-20T03:35:42.383 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5284999999999999e-05 2023-10-20T03:35:42.383 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.384 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.384 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.384 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.372826+0000", 2023-10-20T03:35:42.384 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014215768 2023-10-20T03:35:42.385 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.385 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.385 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.385 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.372884+0000", 2023-10-20T03:35:42.385 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8322000000000002e-05 2023-10-20T03:35:42.386 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.386 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.386 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.386 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.386 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.386 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.6s6 43/31 ECSubWrite(tid=85, reqid=client.4570.0:312, at_version=43'25, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:35:42.387 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:25.374547+0000", 2023-10-20T03:35:42.387 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 16.975369007000001, 2023-10-20T03:35:42.387 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0084348670000000004, 2023-10-20T03:35:42.387 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.387 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.387 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.388 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.388 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.388 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.374547+0000", 2023-10-20T03:35:42.388 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.388 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.389 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.389 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.389 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.374547+0000", 2023-10-20T03:35:42.389 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.389 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.389 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.390 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.390 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.374548+0000", 2023-10-20T03:35:42.390 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.2600000000000001e-07 2023-10-20T03:35:42.390 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.390 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.391 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.391 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.374677+0000", 2023-10-20T03:35:42.391 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012946499999999999 2023-10-20T03:35:42.391 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.391 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.391 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.392 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.374691+0000", 2023-10-20T03:35:42.392 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3959e-05 2023-10-20T03:35:42.392 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.392 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.392 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.392 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.374698+0000", 2023-10-20T03:35:42.393 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0500000000000003e-06 2023-10-20T03:35:42.393 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.393 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.393 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.393 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.374777+0000", 2023-10-20T03:35:42.393 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8794000000000005e-05 2023-10-20T03:35:42.394 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.394 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.394 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.394 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.374789+0000", 2023-10-20T03:35:42.394 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2245999999999999e-05 2023-10-20T03:35:42.395 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.395 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.395 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.395 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.382911+0000", 2023-10-20T03:35:42.395 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0081220730000000005 2023-10-20T03:35:42.395 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.396 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.396 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.396 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.382981+0000", 2023-10-20T03:35:42.396 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0353999999999998e-05 2023-10-20T03:35:42.396 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.397 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.397 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.397 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.397 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.397 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.6s6 43/31 ECSubWrite(tid=88, reqid=client.4570.0:313, at_version=43'26, trim_to=0'0, roll_forward_to=43'24))", 2023-10-20T03:35:42.397 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:25.388156+0000", 2023-10-20T03:35:42.398 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 16.961759954000001, 2023-10-20T03:35:42.398 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0078166539999999993, 2023-10-20T03:35:42.398 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.398 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.398 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.399 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.399 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.399 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.388156+0000", 2023-10-20T03:35:42.399 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.399 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.399 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.400 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.400 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.388156+0000", 2023-10-20T03:35:42.400 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.400 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.400 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.400 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.401 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.388157+0000", 2023-10-20T03:35:42.401 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.55e-06 2023-10-20T03:35:42.401 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.401 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.401 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.401 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.388301+0000", 2023-10-20T03:35:42.402 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014353900000000001 2023-10-20T03:35:42.402 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.402 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.402 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.402 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.388331+0000", 2023-10-20T03:35:42.402 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9901999999999999e-05 2023-10-20T03:35:42.403 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.403 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.403 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.403 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.388346+0000", 2023-10-20T03:35:42.403 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4983e-05 2023-10-20T03:35:42.404 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.404 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.404 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.404 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.388523+0000", 2023-10-20T03:35:42.404 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017784100000000001 2023-10-20T03:35:42.404 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.405 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.405 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.405 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.388553+0000", 2023-10-20T03:35:42.405 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9448999999999999e-05 2023-10-20T03:35:42.405 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.405 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.406 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.406 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.395909+0000", 2023-10-20T03:35:42.406 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0073556910000000001 2023-10-20T03:35:42.406 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.406 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.407 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.407 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.395972+0000", 2023-10-20T03:35:42.407 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3699000000000001e-05 2023-10-20T03:35:42.407 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.407 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.407 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.408 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.408 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.408 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 44/31 ECSubWrite(tid=18, reqid=client.4570.0:323, at_version=44'9, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:35:42.408 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.361265+0000", 2023-10-20T03:35:42.408 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 15.98865106, 2023-10-20T03:35:42.409 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016040175, 2023-10-20T03:35:42.409 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.409 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.409 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.409 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.409 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.410 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361265+0000", 2023-10-20T03:35:42.410 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.410 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.410 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.410 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.411 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361265+0000", 2023-10-20T03:35:42.411 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.411 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.411 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.411 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.412 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361265+0000", 2023-10-20T03:35:42.412 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9999999999999998e-07 2023-10-20T03:35:42.412 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.412 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.412 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.412 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361583+0000", 2023-10-20T03:35:42.413 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00031771700000000002 2023-10-20T03:35:42.413 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.413 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.413 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.413 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361616+0000", 2023-10-20T03:35:42.413 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3555e-05 2023-10-20T03:35:42.414 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.414 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.414 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.414 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361639+0000", 2023-10-20T03:35:42.414 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2238999999999998e-05 2023-10-20T03:35:42.415 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.415 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.415 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.415 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361965+0000", 2023-10-20T03:35:42.415 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00032612299999999999 2023-10-20T03:35:42.415 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.416 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.416 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.416 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361993+0000", 2023-10-20T03:35:42.416 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8535999999999999e-05 2023-10-20T03:35:42.416 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.416 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.417 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.417 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.377210+0000", 2023-10-20T03:35:42.417 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015216644 2023-10-20T03:35:42.417 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.417 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.417 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.418 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.377305+0000", 2023-10-20T03:35:42.418 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4860999999999999e-05 2023-10-20T03:35:42.418 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.418 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.418 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.419 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.419 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.419 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.6s6 44/31 ECSubWrite(tid=94, reqid=client.4570.0:328, at_version=44'31, trim_to=0'0, roll_forward_to=44'29))", 2023-10-20T03:35:42.419 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.372103+0000", 2023-10-20T03:35:42.419 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 15.977812341, 2023-10-20T03:35:42.419 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0078017529999999998, 2023-10-20T03:35:42.420 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.420 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.420 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.420 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.420 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.420 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.372103+0000", 2023-10-20T03:35:42.421 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.421 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.421 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.421 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.421 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.372103+0000", 2023-10-20T03:35:42.421 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.422 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.422 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.422 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.422 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.372105+0000", 2023-10-20T03:35:42.423 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5990000000000001e-06 2023-10-20T03:35:42.423 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.423 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.423 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.423 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.372272+0000", 2023-10-20T03:35:42.424 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016728500000000001 2023-10-20T03:35:42.424 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.424 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.424 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.424 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.372299+0000", 2023-10-20T03:35:42.425 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6605999999999999e-05 2023-10-20T03:35:42.425 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.425 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.425 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.425 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.372316+0000", 2023-10-20T03:35:42.425 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6920000000000001e-05 2023-10-20T03:35:42.426 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.426 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.426 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.426 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.372468+0000", 2023-10-20T03:35:42.426 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015200899999999999 2023-10-20T03:35:42.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.427 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.427 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.427 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.372567+0000", 2023-10-20T03:35:42.427 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9023000000000004e-05 2023-10-20T03:35:42.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.428 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.428 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.428 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.379821+0000", 2023-10-20T03:35:42.428 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0072542769999999999 2023-10-20T03:35:42.428 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.428 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.429 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.429 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.379905+0000", 2023-10-20T03:35:42.429 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4034000000000002e-05 2023-10-20T03:35:42.429 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.429 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.429 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.430 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.430 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.430 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 44/31 ECSubWrite(tid=19, reqid=client.4570.0:324, at_version=44'10, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:35:42.430 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.381682+0000", 2023-10-20T03:35:42.430 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 15.968233230999999, 2023-10-20T03:35:42.431 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013179592, 2023-10-20T03:35:42.431 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.431 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.431 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.431 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.431 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.432 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.381682+0000", 2023-10-20T03:35:42.432 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.432 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.432 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.432 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.432 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.381682+0000", 2023-10-20T03:35:42.433 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.433 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.433 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.433 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.433 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.381684+0000", 2023-10-20T03:35:42.433 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.897e-06 2023-10-20T03:35:42.434 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.434 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.434 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.434 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.382005+0000", 2023-10-20T03:35:42.434 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000320649 2023-10-20T03:35:42.435 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.435 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.435 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.435 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.382041+0000", 2023-10-20T03:35:42.435 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6242000000000003e-05 2023-10-20T03:35:42.436 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.436 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.436 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.436 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.382067+0000", 2023-10-20T03:35:42.436 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6208999999999999e-05 2023-10-20T03:35:42.436 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.437 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.437 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.437 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.382198+0000", 2023-10-20T03:35:42.437 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013078300000000001 2023-10-20T03:35:42.437 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.437 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.438 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.438 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.382221+0000", 2023-10-20T03:35:42.438 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2858999999999999e-05 2023-10-20T03:35:42.438 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.438 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.439 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.439 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.394748+0000", 2023-10-20T03:35:42.439 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012526822999999999 2023-10-20T03:35:42.439 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.439 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.439 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.440 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.394862+0000", 2023-10-20T03:35:42.440 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011413000000000001 2023-10-20T03:35:42.440 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.440 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.440 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.441 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.441 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 49/31 ECSubWrite(tid=39, reqid=client.4570.0:390, at_version=49'23, trim_to=0'0, roll_forward_to=47'21))", 2023-10-20T03:35:42.441 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:31.415158+0000", 2023-10-20T03:35:42.441 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 10.934757551000001, 2023-10-20T03:35:42.441 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.020096919000000001, 2023-10-20T03:35:42.442 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.442 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.442 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.442 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.442 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.443 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415158+0000", 2023-10-20T03:35:42.443 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.443 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.443 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.443 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.443 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415158+0000", 2023-10-20T03:35:42.444 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.444 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.444 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.444 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415161+0000", 2023-10-20T03:35:42.445 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6690000000000002e-06 2023-10-20T03:35:42.445 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.445 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.445 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.445 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415623+0000", 2023-10-20T03:35:42.446 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00046182100000000001 2023-10-20T03:35:42.446 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.446 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.446 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.446 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415659+0000", 2023-10-20T03:35:42.446 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6779000000000001e-05 2023-10-20T03:35:42.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.447 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.447 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.447 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415678+0000", 2023-10-20T03:35:42.447 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8635999999999999e-05 2023-10-20T03:35:42.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.448 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.448 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.448 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415832+0000", 2023-10-20T03:35:42.448 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015418300000000001 2023-10-20T03:35:42.448 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.448 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.449 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.449 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415848+0000", 2023-10-20T03:35:42.449 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5506e-05 2023-10-20T03:35:42.449 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.449 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.449 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.450 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.435162+0000", 2023-10-20T03:35:42.450 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019313974000000001 2023-10-20T03:35:42.450 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.450 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.450 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.450 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.435255+0000", 2023-10-20T03:35:42.451 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3350999999999994e-05 2023-10-20T03:35:42.451 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.451 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.451 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.451 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.452 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.452 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 53/47 ECSubWrite(tid=56, reqid=osd.3.0:55, at_version=0'0, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:42.452 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.448564+0000", 2023-10-20T03:35:42.452 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 6.9013514770000004, 2023-10-20T03:35:42.452 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014906802, 2023-10-20T03:35:42.452 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.453 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.453 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.453 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.453 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.453 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448564+0000", 2023-10-20T03:35:42.454 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.454 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.454 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.454 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.454 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448564+0000", 2023-10-20T03:35:42.455 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.455 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.455 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.455 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.455 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448566+0000", 2023-10-20T03:35:42.455 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8139999999999999e-06 2023-10-20T03:35:42.456 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.456 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.456 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.456 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448593+0000", 2023-10-20T03:35:42.456 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6829e-05 2023-10-20T03:35:42.456 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.457 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.457 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.457 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448629+0000", 2023-10-20T03:35:42.457 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6449000000000003e-05 2023-10-20T03:35:42.457 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.458 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.458 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.458 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448647+0000", 2023-10-20T03:35:42.458 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7895999999999999e-05 2023-10-20T03:35:42.458 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.458 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.459 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.459 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448792+0000", 2023-10-20T03:35:42.459 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014443400000000001 2023-10-20T03:35:42.459 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.459 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.459 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.460 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448808+0000", 2023-10-20T03:35:42.460 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6602000000000001e-05 2023-10-20T03:35:42.460 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.460 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.460 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.461 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.463361+0000", 2023-10-20T03:35:42.461 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0145524 2023-10-20T03:35:42.461 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.461 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.461 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.461 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.463471+0000", 2023-10-20T03:35:42.462 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011037800000000001 2023-10-20T03:35:42.462 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.462 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.462 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.462 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.462 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.463 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 53/47 ECSubWrite(tid=57, reqid=client.4570.0:408, at_version=53'31, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:42.463 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.449547+0000", 2023-10-20T03:35:42.463 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 6.9003690710000001, 2023-10-20T03:35:42.463 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015260123, 2023-10-20T03:35:42.463 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.464 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.464 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.464 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.464 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.464 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.449547+0000", 2023-10-20T03:35:42.464 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.465 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.465 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.465 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.465 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.449547+0000", 2023-10-20T03:35:42.465 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.465 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.466 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.466 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.466 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.449548+0000", 2023-10-20T03:35:42.466 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.767e-06 2023-10-20T03:35:42.466 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.466 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.467 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.467 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.449997+0000", 2023-10-20T03:35:42.467 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00044844900000000002 2023-10-20T03:35:42.467 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.467 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.468 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.468 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.450025+0000", 2023-10-20T03:35:42.468 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7826000000000001e-05 2023-10-20T03:35:42.468 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.468 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.468 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.469 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.450038+0000", 2023-10-20T03:35:42.469 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3383e-05 2023-10-20T03:35:42.469 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.469 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.469 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.469 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.450200+0000", 2023-10-20T03:35:42.470 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000162349 2023-10-20T03:35:42.470 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.470 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.470 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.470 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.450221+0000", 2023-10-20T03:35:42.471 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1078999999999999e-05 2023-10-20T03:35:42.471 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.471 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.471 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.471 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.464734+0000", 2023-10-20T03:35:42.471 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014512771000000001 2023-10-20T03:35:42.472 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.472 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.472 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.472 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.464807+0000", 2023-10-20T03:35:42.472 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2498999999999998e-05 2023-10-20T03:35:42.473 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.473 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.473 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.473 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.473 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.474 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 53/47 ECSubWrite(tid=58, reqid=client.4570.0:409, at_version=53'32, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:42.474 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.473255+0000", 2023-10-20T03:35:42.474 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 6.8766601200000004, 2023-10-20T03:35:42.474 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011057342, 2023-10-20T03:35:42.474 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.475 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.475 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.475 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.475 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.475 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473255+0000", 2023-10-20T03:35:42.475 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.476 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.476 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.476 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.476 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473255+0000", 2023-10-20T03:35:42.476 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.477 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.477 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.477 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.477 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473257+0000", 2023-10-20T03:35:42.477 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9069999999999999e-06 2023-10-20T03:35:42.477 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.478 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.478 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.478 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473620+0000", 2023-10-20T03:35:42.478 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000362667 2023-10-20T03:35:42.478 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.479 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.479 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.479 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473671+0000", 2023-10-20T03:35:42.479 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0828999999999997e-05 2023-10-20T03:35:42.479 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.479 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.480 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.480 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473698+0000", 2023-10-20T03:35:42.480 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6783000000000001e-05 2023-10-20T03:35:42.480 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.480 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.480 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.481 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473830+0000", 2023-10-20T03:35:42.481 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013184000000000001 2023-10-20T03:35:42.481 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.481 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.481 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.482 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473851+0000", 2023-10-20T03:35:42.482 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1716999999999999e-05 2023-10-20T03:35:42.482 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.482 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.482 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.483 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.484230+0000", 2023-10-20T03:35:42.483 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010379256 2023-10-20T03:35:42.483 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.483 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.483 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.483 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.484313+0000", 2023-10-20T03:35:42.484 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2342999999999994e-05 2023-10-20T03:35:42.484 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.484 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.484 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.484 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.485 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.485 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 54/31 ECSubWrite(tid=65, reqid=client.4570.0:418, at_version=54'32, trim_to=0'0, roll_forward_to=51'31))", 2023-10-20T03:35:42.485 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.459426+0000", 2023-10-20T03:35:42.485 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 5.8904892059999998, 2023-10-20T03:35:42.485 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015193442999999999, 2023-10-20T03:35:42.485 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.486 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.486 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.486 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.486 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.486 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459426+0000", 2023-10-20T03:35:42.487 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.487 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.487 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.487 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.487 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459426+0000", 2023-10-20T03:35:42.487 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.488 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.488 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.488 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.488 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459429+0000", 2023-10-20T03:35:42.488 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2900000000000001e-06 2023-10-20T03:35:42.488 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.489 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.489 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.489 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459850+0000", 2023-10-20T03:35:42.489 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000421266 2023-10-20T03:35:42.489 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.489 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.490 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.490 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459900+0000", 2023-10-20T03:35:42.490 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9914999999999999e-05 2023-10-20T03:35:42.490 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.490 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.491 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.491 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459925+0000", 2023-10-20T03:35:42.491 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5596e-05 2023-10-20T03:35:42.491 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.491 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.491 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.492 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.460072+0000", 2023-10-20T03:35:42.492 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014694400000000001 2023-10-20T03:35:42.492 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.492 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.492 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.493 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.460100+0000", 2023-10-20T03:35:42.493 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7087999999999999e-05 2023-10-20T03:35:42.493 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.493 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.493 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.494 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.474528+0000", 2023-10-20T03:35:42.494 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014428806000000001 2023-10-20T03:35:42.494 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.494 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.494 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.494 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.474620+0000", 2023-10-20T03:35:42.495 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1538000000000003e-05 2023-10-20T03:35:42.495 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.495 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.495 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.495 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.496 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.496 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 54/31 ECSubWrite(tid=66, reqid=client.4570.0:419, at_version=54'33, trim_to=0'0, roll_forward_to=51'31))", 2023-10-20T03:35:42.496 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.476272+0000", 2023-10-20T03:35:42.496 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 5.8736431790000001, 2023-10-20T03:35:42.496 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010573028, 2023-10-20T03:35:42.496 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.497 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.497 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.497 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.497 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.497 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476272+0000", 2023-10-20T03:35:42.497 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.498 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.498 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.498 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.498 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476272+0000", 2023-10-20T03:35:42.498 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.499 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.499 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.499 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.499 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476274+0000", 2023-10-20T03:35:42.499 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9130000000000001e-06 2023-10-20T03:35:42.499 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.500 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.500 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.500 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476566+0000", 2023-10-20T03:35:42.500 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00029126899999999998 2023-10-20T03:35:42.500 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.500 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.501 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.501 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476615+0000", 2023-10-20T03:35:42.501 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9188999999999999e-05 2023-10-20T03:35:42.501 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.501 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.501 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.502 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476640+0000", 2023-10-20T03:35:42.502 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5128999999999999e-05 2023-10-20T03:35:42.502 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.502 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.502 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.503 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476792+0000", 2023-10-20T03:35:42.503 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015181 2023-10-20T03:35:42.503 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.503 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.503 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.504 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476808+0000", 2023-10-20T03:35:42.504 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5882000000000001e-05 2023-10-20T03:35:42.504 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.504 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.504 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.504 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.486770+0000", 2023-10-20T03:35:42.505 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0099620839999999995 2023-10-20T03:35:42.505 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.505 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.505 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.505 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.486845+0000", 2023-10-20T03:35:42.506 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5752000000000002e-05 2023-10-20T03:35:42.506 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.506 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.506 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.506 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.506 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.507 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 54/31 ECSubWrite(tid=67, reqid=client.4570.0:420, at_version=54'34, trim_to=0'0, roll_forward_to=54'32))", 2023-10-20T03:35:42.507 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.488419+0000", 2023-10-20T03:35:42.507 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 5.8614965950000002, 2023-10-20T03:35:42.507 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0089196740000000007, 2023-10-20T03:35:42.507 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.508 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.508 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.508 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.508 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.508 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488419+0000", 2023-10-20T03:35:42.508 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.509 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.509 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.509 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.509 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488419+0000", 2023-10-20T03:35:42.509 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.509 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.510 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.510 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.510 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488421+0000", 2023-10-20T03:35:42.510 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8899999999999999e-06 2023-10-20T03:35:42.510 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.511 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.511 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.511 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488517+0000", 2023-10-20T03:35:42.511 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6211000000000004e-05 2023-10-20T03:35:42.511 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.511 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.512 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.512 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488591+0000", 2023-10-20T03:35:42.512 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3850999999999995e-05 2023-10-20T03:35:42.512 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.512 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.513 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.513 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488613+0000", 2023-10-20T03:35:42.513 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1985999999999999e-05 2023-10-20T03:35:42.513 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.513 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.513 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.514 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488771+0000", 2023-10-20T03:35:42.514 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000158386 2023-10-20T03:35:42.514 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.514 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.514 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.515 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488799+0000", 2023-10-20T03:35:42.515 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7503000000000001e-05 2023-10-20T03:35:42.515 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.515 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.515 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.516 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.497130+0000", 2023-10-20T03:35:42.516 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0083314960000000007 2023-10-20T03:35:42.516 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.516 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.516 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.516 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.497339+0000", 2023-10-20T03:35:42.517 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020835100000000001 2023-10-20T03:35:42.517 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.517 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.517 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.517 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.518 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.518 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 55/47 ECSubWrite(tid=71, reqid=client.4570.0:406, at_version=55'38, trim_to=0'0, roll_forward_to=54'36))", 2023-10-20T03:35:42.518 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:37.473148+0000", 2023-10-20T03:35:42.518 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4.8767677909999998, 2023-10-20T03:35:42.518 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013993848, 2023-10-20T03:35:42.519 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.519 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.519 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.519 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.519 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.519 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473148+0000", 2023-10-20T03:35:42.520 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.520 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.520 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.520 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.520 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473148+0000", 2023-10-20T03:35:42.521 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.521 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.521 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.521 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.521 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473148+0000", 2023-10-20T03:35:42.521 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6600000000000002e-07 2023-10-20T03:35:42.522 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.522 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.522 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.522 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473628+0000", 2023-10-20T03:35:42.523 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00047980800000000002 2023-10-20T03:35:42.523 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.523 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.523 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.523 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473658+0000", 2023-10-20T03:35:42.524 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0159999999999999e-05 2023-10-20T03:35:42.524 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.524 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.524 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.524 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473681+0000", 2023-10-20T03:35:42.525 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2433e-05 2023-10-20T03:35:42.525 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.525 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.525 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.525 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473843+0000", 2023-10-20T03:35:42.526 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016257500000000001 2023-10-20T03:35:42.526 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.526 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.526 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.526 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473860+0000", 2023-10-20T03:35:42.526 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6543999999999999e-05 2023-10-20T03:35:42.527 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.527 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.527 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.527 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.487051+0000", 2023-10-20T03:35:42.527 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013190947 2023-10-20T03:35:42.527 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.528 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.528 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.528 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.487142+0000", 2023-10-20T03:35:42.528 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1014999999999995e-05 2023-10-20T03:35:42.528 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.528 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.529 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.529 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.529 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.529 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/47 ECSubWrite(tid=83, reqid=osd.3.0:83, at_version=60'48, trim_to=0'0, roll_forward_to=56'46))", 2023-10-20T03:35:42.529 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.854857+0000", 2023-10-20T03:35:42.529 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.495058742, 2023-10-20T03:35:42.530 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012200993, 2023-10-20T03:35:42.530 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.530 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.530 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.530 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.531 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.531 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.854857+0000", 2023-10-20T03:35:42.531 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.531 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.531 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.531 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.532 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.854857+0000", 2023-10-20T03:35:42.532 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.532 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.532 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.532 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.532 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.854858+0000", 2023-10-20T03:35:42.533 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9299999999999997e-07 2023-10-20T03:35:42.533 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.533 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.533 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.533 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.854919+0000", 2023-10-20T03:35:42.534 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1148999999999993e-05 2023-10-20T03:35:42.534 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.534 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.534 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.534 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.855027+0000", 2023-10-20T03:35:42.534 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010831399999999999 2023-10-20T03:35:42.535 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.535 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.535 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.535 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.855057+0000", 2023-10-20T03:35:42.535 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9396000000000001e-05 2023-10-20T03:35:42.536 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.536 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.536 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.536 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.855236+0000", 2023-10-20T03:35:42.536 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000179366 2023-10-20T03:35:42.536 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.537 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.537 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.537 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.855267+0000", 2023-10-20T03:35:42.537 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.1133000000000002e-05 2023-10-20T03:35:42.537 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.537 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.538 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.538 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.866928+0000", 2023-10-20T03:35:42.538 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011660841 2023-10-20T03:35:42.538 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.538 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.539 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.539 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.867058+0000", 2023-10-20T03:35:42.539 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000130001 2023-10-20T03:35:42.539 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.539 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.539 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.540 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.540 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.540 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/47 ECSubWrite(tid=84, reqid=osd.3.0:84, at_version=60'50, trim_to=0'0, roll_forward_to=56'46))", 2023-10-20T03:35:42.540 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.857484+0000", 2023-10-20T03:35:42.540 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.49243161600000002, 2023-10-20T03:35:42.540 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012226756, 2023-10-20T03:35:42.541 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.541 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.541 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.541 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.541 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.541 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857484+0000", 2023-10-20T03:35:42.542 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.542 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.542 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.542 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.542 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857484+0000", 2023-10-20T03:35:42.542 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.543 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.543 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.543 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.543 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857486+0000", 2023-10-20T03:35:42.543 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.04e-06 2023-10-20T03:35:42.544 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.544 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.544 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.544 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857540+0000", 2023-10-20T03:35:42.544 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4339000000000003e-05 2023-10-20T03:35:42.545 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.545 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.545 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.545 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857575+0000", 2023-10-20T03:35:42.545 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4555999999999999e-05 2023-10-20T03:35:42.545 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.546 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.546 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.546 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857598+0000", 2023-10-20T03:35:42.546 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3193e-05 2023-10-20T03:35:42.546 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.547 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.547 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.547 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.864569+0000", 2023-10-20T03:35:42.547 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0069706580000000002 2023-10-20T03:35:42.547 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.548 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.548 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.548 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.864611+0000", 2023-10-20T03:35:42.548 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1909000000000001e-05 2023-10-20T03:35:42.548 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.548 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.549 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.549 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.869616+0000", 2023-10-20T03:35:42.549 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00500557 2023-10-20T03:35:42.549 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.549 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.550 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.550 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.869711+0000", 2023-10-20T03:35:42.550 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4491e-05 2023-10-20T03:35:42.550 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.550 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.550 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.551 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.551 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.551 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/47 ECSubWrite(tid=86, reqid=osd.3.0:86, at_version=60'52, trim_to=0'0, roll_forward_to=60'50))", 2023-10-20T03:35:42.551 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.888525+0000", 2023-10-20T03:35:42.551 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.46139095099999999, 2023-10-20T03:35:42.551 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0082370510000000004, 2023-10-20T03:35:42.552 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.552 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.552 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.552 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.552 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.553 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888525+0000", 2023-10-20T03:35:42.553 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.553 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.553 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.553 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.553 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888525+0000", 2023-10-20T03:35:42.554 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.554 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.554 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.554 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.554 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888527+0000", 2023-10-20T03:35:42.555 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9740000000000001e-06 2023-10-20T03:35:42.555 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.555 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.555 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.555 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888587+0000", 2023-10-20T03:35:42.556 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0225e-05 2023-10-20T03:35:42.556 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.556 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.556 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.556 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888636+0000", 2023-10-20T03:35:42.556 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8959000000000003e-05 2023-10-20T03:35:42.557 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.557 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.557 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.557 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888665+0000", 2023-10-20T03:35:42.557 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9403e-05 2023-10-20T03:35:42.557 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.558 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.558 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.558 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888841+0000", 2023-10-20T03:35:42.558 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000175482 2023-10-20T03:35:42.558 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.558 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.559 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.559 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888880+0000", 2023-10-20T03:35:42.559 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9393e-05 2023-10-20T03:35:42.559 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.559 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.560 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.560 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.896686+0000", 2023-10-20T03:35:42.560 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0078062660000000001 2023-10-20T03:35:42.560 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.560 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.560 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.561 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.896762+0000", 2023-10-20T03:35:42.561 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5349e-05 2023-10-20T03:35:42.561 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.561 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.561 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.561 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.562 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.562 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/47 ECSubWrite(tid=87, reqid=osd.3.0:87, at_version=60'54, trim_to=0'0, roll_forward_to=60'50))", 2023-10-20T03:35:42.562 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.888724+0000", 2023-10-20T03:35:42.562 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.46119122200000001, 2023-10-20T03:35:42.562 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010164182000000001, 2023-10-20T03:35:42.563 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:42.563 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:42.563 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:42.563 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.563 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:42.564 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888724+0000", 2023-10-20T03:35:42.564 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.564 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.564 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.564 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:42.564 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888724+0000", 2023-10-20T03:35:42.565 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:42.565 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.565 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.565 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:42.565 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888725+0000", 2023-10-20T03:35:42.566 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0240000000000001e-06 2023-10-20T03:35:42.566 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.566 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.566 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:42.566 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888808+0000", 2023-10-20T03:35:42.566 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2757999999999996e-05 2023-10-20T03:35:42.567 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.567 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.567 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:42.567 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888836+0000", 2023-10-20T03:35:42.567 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7572e-05 2023-10-20T03:35:42.567 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.568 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.568 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:42.568 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888855+0000", 2023-10-20T03:35:42.568 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.961e-05 2023-10-20T03:35:42.568 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.569 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.569 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:42.569 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889902+0000", 2023-10-20T03:35:42.569 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0010469730000000001 2023-10-20T03:35:42.569 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.569 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.570 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:42.570 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889928+0000", 2023-10-20T03:35:42.570 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5916000000000001e-05 2023-10-20T03:35:42.570 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.570 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.571 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:42.571 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.898832+0000", 2023-10-20T03:35:42.571 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0089040820000000007 2023-10-20T03:35:42.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:42.571 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:42.571 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:42.572 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.898889+0000", 2023-10-20T03:35:42.572 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6246999999999999e-05 2023-10-20T03:35:42.572 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.572 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.572 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.573 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:42.573 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:42.573 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:42.574 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:42.575 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.575 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:42.576 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.576 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-10-20T03:35:42.577 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:42.577 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.578 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:42.572+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:42.584 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:42.585 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.614 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:42.614 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.646 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:42.646 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.646 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:42.667 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:42.667 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.676 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:42.674+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:42.683 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:42.684 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.704 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:42.704 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:42.709 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:42.709 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.719 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-10-20T03:35:42.732 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:42.733 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.756 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:42.757 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.776 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:42.772+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:42.777 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:42.777 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.811 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:42.812 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.833 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:42.833 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.846 INFO:tasks.rados.rados.0.smithi170.stdout:200: done (6 left) 2023-10-20T03:35:42.846 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 19 current snap is 13 2023-10-20T03:35:42.846 INFO:tasks.rados.rados.0.smithi170.stdout:201: seq_num 98 ranges {0=491520,491520=589824,1081344=557056,1638400=16384} 2023-10-20T03:35:42.849 INFO:tasks.rados.rados.0.smithi170.stdout:201: writing smithi17039300-19 from 0 to 491520 tid 1 2023-10-20T03:35:42.854 INFO:tasks.rados.rados.0.smithi170.stdout:201: writing smithi17039300-19 from 491520 to 1081344 tid 2 2023-10-20T03:35:42.859 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:42.859 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:35:42.860 INFO:tasks.rados.rados.0.smithi170.stdout:201: writing smithi17039300-19 from 1081344 to 1638400 tid 3 2023-10-20T03:35:42.860 INFO:tasks.rados.rados.0.smithi170.stdout:201: writing smithi17039300-19 from 1638400 to 1654784 tid 4 2023-10-20T03:35:42.861 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 14 current snap is 13 2023-10-20T03:35:42.861 INFO:tasks.rados.rados.0.smithi170.stdout:202: seq_num 99 ranges {3031040=507904,3538944=606208,4145152=557056} 2023-10-20T03:35:42.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:42.864 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:42.866 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:42.866 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:42.877 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:42.872+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:42.881 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-10-20T03:35:42.888 INFO:tasks.rados.rados.0.smithi170.stdout:202: writing smithi17039300-14 from 3031040 to 3538944 tid 1 2023-10-20T03:35:42.893 INFO:tasks.rados.rados.0.smithi170.stdout:202: writing smithi17039300-14 from 3538944 to 4145152 tid 2 2023-10-20T03:35:42.898 INFO:tasks.rados.rados.0.smithi170.stdout:202: writing smithi17039300-14 from 4145152 to 4702208 tid 3 2023-10-20T03:35:42.898 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 25 current snap is 13 2023-10-20T03:35:42.898 INFO:tasks.rados.rados.0.smithi170.stdout:193: finishing write tid 1 to smithi17039300-28 2023-10-20T03:35:42.898 INFO:tasks.rados.rados.0.smithi170.stdout:203: seq_num 100 ranges {802816=524288,1327104=753664,2080768=409600} 2023-10-20T03:35:42.909 INFO:tasks.rados.rados.0.smithi170.stdout:203: writing smithi17039300-25 from 802816 to 1327104 tid 1 2023-10-20T03:35:42.915 INFO:tasks.rados.rados.0.smithi170.stdout:203: writing smithi17039300-25 from 1327104 to 2080768 tid 2 2023-10-20T03:35:42.919 INFO:tasks.rados.rados.0.smithi170.stdout:203: writing smithi17039300-25 from 2080768 to 2490368 tid 3 2023-10-20T03:35:42.919 INFO:tasks.rados.rados.0.smithi170.stdout:204: rmattr oid 46 current snap is 13 2023-10-20T03:35:42.920 INFO:tasks.rados.rados.0.smithi170.stdout:204: done (9 left) 2023-10-20T03:35:42.920 INFO:tasks.rados.rados.0.smithi170.stdout:205: read oid 15 snap -1 2023-10-20T03:35:42.920 INFO:tasks.rados.rados.0.smithi170.stdout:205: expect (ObjNum 14 snap 0 seq_num 14) 2023-10-20T03:35:42.920 INFO:tasks.rados.rados.0.smithi170.stdout:206: read oid 45 snap -1 2023-10-20T03:35:42.920 INFO:tasks.rados.rados.0.smithi170.stdout:206: expect (ObjNum 44 snap 0 seq_num 44) 2023-10-20T03:35:42.921 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 33 current snap is 13 2023-10-20T03:35:42.921 INFO:tasks.rados.rados.0.smithi170.stdout:207: seq_num 101 ranges {884736=540672,1425408=507904,1933312=622592,2555904=32768} 2023-10-20T03:35:42.930 INFO:tasks.rados.rados.0.smithi170.stdout:207: writing smithi17039300-33 from 884736 to 1425408 tid 1 2023-10-20T03:35:42.933 INFO:tasks.rados.rados.0.smithi170.stdout:207: writing smithi17039300-33 from 1425408 to 1933312 tid 2 2023-10-20T03:35:42.937 INFO:tasks.rados.rados.0.smithi170.stdout:207: writing smithi17039300-33 from 1933312 to 2555904 tid 3 2023-10-20T03:35:42.938 INFO:tasks.rados.rados.0.smithi170.stdout:207: writing smithi17039300-33 from 2555904 to 2588672 tid 4 2023-10-20T03:35:42.938 INFO:tasks.rados.rados.0.smithi170.stdout:208: delete oid 13 current snap is 13 2023-10-20T03:35:42.938 INFO:tasks.rados.rados.0.smithi170.stdout:196: expect (ObjNum 39 snap 0 seq_num 39) 2023-10-20T03:35:42.978 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:42.974+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:42.991 INFO:tasks.rados.rados.0.smithi170.stdout:193: finishing write tid 2 to smithi17039300-28 2023-10-20T03:35:42.992 INFO:tasks.rados.rados.0.smithi170.stdout:193: finishing write tid 3 to smithi17039300-28 2023-10-20T03:35:42.992 INFO:tasks.rados.rados.0.smithi170.stdout:193: finishing write tid 4 to smithi17039300-28 2023-10-20T03:35:42.992 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 20 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-10-20T03:35:42.992 INFO:tasks.rados.rados.0.smithi170.stdout:195: finishing write tid 1 to smithi17039300-38 2023-10-20T03:35:42.992 INFO:tasks.rados.rados.0.smithi170.stdout:193: finishing write tid 5 to smithi17039300-28 2023-10-20T03:35:42.993 INFO:tasks.rados.rados.0.smithi170.stdout:193: oid 28 updating version 0 to 54 2023-10-20T03:35:42.993 INFO:tasks.rados.rados.0.smithi170.stdout:193: oid 28 updating version 54 to 57 2023-10-20T03:35:42.993 INFO:tasks.rados.rados.0.smithi170.stdout:193: oid 28 version 57 is already newer than 56 2023-10-20T03:35:42.993 INFO:tasks.rados.rados.0.smithi170.stdout:193: oid 28 version 57 is already newer than 55 2023-10-20T03:35:42.993 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 28 v 57 (ObjNum 94 snap 13 seq_num 94) dirty exists 2023-10-20T03:35:42.993 INFO:tasks.rados.rados.0.smithi170.stdout:193: left oid 28 (ObjNum 94 snap 13 seq_num 94) 2023-10-20T03:35:42.994 INFO:tasks.rados.rados.0.smithi170.stdout:195: finishing write tid 2 to smithi17039300-38 2023-10-20T03:35:42.994 INFO:tasks.rados.rados.0.smithi170.stdout:198: expect (ObjNum 77 snap 5 seq_num 77) 2023-10-20T03:35:43.038 INFO:tasks.rados.rados.0.smithi170.stdout:195: finishing write tid 3 to smithi17039300-38 2023-10-20T03:35:43.039 INFO:tasks.rados.rados.0.smithi170.stdout:195: finishing write tid 4 to smithi17039300-38 2023-10-20T03:35:43.039 INFO:tasks.rados.rados.0.smithi170.stdout:195: finishing write tid 5 to smithi17039300-38 2023-10-20T03:35:43.041 INFO:tasks.rados.rados.0.smithi170.stdout:195: oid 38 updating version 0 to 9 2023-10-20T03:35:43.041 INFO:tasks.rados.rados.0.smithi170.stdout:195: oid 38 updating version 9 to 10 2023-10-20T03:35:43.041 INFO:tasks.rados.rados.0.smithi170.stdout:195: oid 38 version 10 is already newer than 8 2023-10-20T03:35:43.041 INFO:tasks.rados.rados.0.smithi170.stdout:195: oid 38 updating version 10 to 11 2023-10-20T03:35:43.042 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 38 v 11 (ObjNum 96 snap 13 seq_num 96) dirty exists 2023-10-20T03:35:43.042 INFO:tasks.rados.rados.0.smithi170.stdout:195: left oid 38 (ObjNum 96 snap 13 seq_num 96) 2023-10-20T03:35:43.042 INFO:tasks.rados.rados.0.smithi170.stdout:199: finishing write tid 1 to smithi17039300-11 2023-10-20T03:35:43.042 INFO:tasks.rados.rados.0.smithi170.stdout:199: finishing write tid 2 to smithi17039300-11 2023-10-20T03:35:43.042 INFO:tasks.rados.rados.0.smithi170.stdout:199: finishing write tid 3 to smithi17039300-11 2023-10-20T03:35:43.043 INFO:tasks.rados.rados.0.smithi170.stdout:199: finishing write tid 4 to smithi17039300-11 2023-10-20T03:35:43.043 INFO:tasks.rados.rados.0.smithi170.stdout:199: finishing write tid 5 to smithi17039300-11 2023-10-20T03:35:43.043 INFO:tasks.rados.rados.0.smithi170.stdout:199: finishing write tid 6 to smithi17039300-11 2023-10-20T03:35:43.043 INFO:tasks.rados.rados.0.smithi170.stdout:199: oid 11 updating version 0 to 25 2023-10-20T03:35:43.043 INFO:tasks.rados.rados.0.smithi170.stdout:199: oid 11 updating version 25 to 26 2023-10-20T03:35:43.044 INFO:tasks.rados.rados.0.smithi170.stdout:199: oid 11 updating version 26 to 27 2023-10-20T03:35:43.044 INFO:tasks.rados.rados.0.smithi170.stdout:199: oid 11 version 27 is already newer than 24 2023-10-20T03:35:43.044 INFO:tasks.rados.rados.0.smithi170.stdout:199: oid 11 updating version 27 to 28 2023-10-20T03:35:43.044 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 11 v 28 (ObjNum 97 snap 13 seq_num 97) dirty exists 2023-10-20T03:35:43.044 INFO:tasks.rados.rados.0.smithi170.stdout:199: left oid 11 (ObjNum 97 snap 13 seq_num 97) 2023-10-20T03:35:43.045 INFO:tasks.rados.rados.0.smithi170.stdout:201: finishing write tid 1 to smithi17039300-19 2023-10-20T03:35:43.045 INFO:tasks.rados.rados.0.smithi170.stdout:201: finishing write tid 2 to smithi17039300-19 2023-10-20T03:35:43.045 INFO:tasks.rados.rados.0.smithi170.stdout:201: finishing write tid 3 to smithi17039300-19 2023-10-20T03:35:43.045 INFO:tasks.rados.rados.0.smithi170.stdout:201: finishing write tid 4 to smithi17039300-19 2023-10-20T03:35:43.045 INFO:tasks.rados.rados.0.smithi170.stdout:201: finishing write tid 5 to smithi17039300-19 2023-10-20T03:35:43.045 INFO:tasks.rados.rados.0.smithi170.stdout:201: finishing write tid 6 to smithi17039300-19 2023-10-20T03:35:43.046 INFO:tasks.rados.rados.0.smithi170.stdout:201: oid 19 updating version 0 to 23 2023-10-20T03:35:43.046 INFO:tasks.rados.rados.0.smithi170.stdout:201: oid 19 updating version 23 to 24 2023-10-20T03:35:43.046 INFO:tasks.rados.rados.0.smithi170.stdout:201: oid 19 updating version 24 to 25 2023-10-20T03:35:43.046 INFO:tasks.rados.rados.0.smithi170.stdout:201: oid 19 version 25 is already newer than 21 2023-10-20T03:35:43.046 INFO:tasks.rados.rados.0.smithi170.stdout:201: oid 19 version 25 is already newer than 22 2023-10-20T03:35:43.046 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 25 (ObjNum 98 snap 13 seq_num 98) dirty exists 2023-10-20T03:35:43.047 INFO:tasks.rados.rados.0.smithi170.stdout:201: left oid 19 (ObjNum 98 snap 13 seq_num 98) 2023-10-20T03:35:43.047 INFO:tasks.rados.rados.0.smithi170.stdout:205: expect (ObjNum 14 snap 0 seq_num 14) 2023-10-20T03:35:43.047 INFO:tasks.rados.rados.0.smithi170.stdout:206: expect (ObjNum 44 snap 0 seq_num 44) 2023-10-20T03:35:43.056 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:43.057 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:43.065 INFO:tasks.rados.rados.0.smithi170.stdout:202: finishing write tid 1 to smithi17039300-14 2023-10-20T03:35:43.065 INFO:tasks.rados.rados.0.smithi170.stdout:203: finishing write tid 1 to smithi17039300-25 2023-10-20T03:35:43.065 INFO:tasks.rados.rados.0.smithi170.stdout:203: finishing write tid 2 to smithi17039300-25 2023-10-20T03:35:43.065 INFO:tasks.rados.rados.0.smithi170.stdout:202: finishing write tid 2 to smithi17039300-14 2023-10-20T03:35:43.066 INFO:tasks.rados.rados.0.smithi170.stdout:203: finishing write tid 3 to smithi17039300-25 2023-10-20T03:35:43.066 INFO:tasks.rados.rados.0.smithi170.stdout:203: finishing write tid 4 to smithi17039300-25 2023-10-20T03:35:43.066 INFO:tasks.rados.rados.0.smithi170.stdout:203: finishing write tid 5 to smithi17039300-25 2023-10-20T03:35:43.066 INFO:tasks.rados.rados.0.smithi170.stdout:203: oid 25 updating version 0 to 34 2023-10-20T03:35:43.066 INFO:tasks.rados.rados.0.smithi170.stdout:203: oid 25 version 34 is already newer than 31 2023-10-20T03:35:43.067 INFO:tasks.rados.rados.0.smithi170.stdout:203: oid 25 version 34 is already newer than 32 2023-10-20T03:35:43.067 INFO:tasks.rados.rados.0.smithi170.stdout:203: oid 25 version 34 is already newer than 33 2023-10-20T03:35:43.067 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 34 (ObjNum 100 snap 13 seq_num 100) dirty exists 2023-10-20T03:35:43.067 INFO:tasks.rados.rados.0.smithi170.stdout:203: left oid 25 (ObjNum 100 snap 13 seq_num 100) 2023-10-20T03:35:43.067 INFO:tasks.rados.rados.0.smithi170.stdout:202: finishing write tid 3 to smithi17039300-14 2023-10-20T03:35:43.067 INFO:tasks.rados.rados.0.smithi170.stdout:202: finishing write tid 4 to smithi17039300-14 2023-10-20T03:35:43.068 INFO:tasks.rados.rados.0.smithi170.stdout:202: finishing write tid 5 to smithi17039300-14 2023-10-20T03:35:43.068 INFO:tasks.rados.rados.0.smithi170.stdout:202: oid 14 updating version 0 to 45 2023-10-20T03:35:43.068 INFO:tasks.rados.rados.0.smithi170.stdout:202: oid 14 updating version 45 to 46 2023-10-20T03:35:43.068 INFO:tasks.rados.rados.0.smithi170.stdout:202: oid 14 updating version 46 to 47 2023-10-20T03:35:43.068 INFO:tasks.rados.rados.0.smithi170.stdout:202: oid 14 version 47 is already newer than 44 2023-10-20T03:35:43.068 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 14 v 47 (ObjNum 99 snap 13 seq_num 99) dirty exists 2023-10-20T03:35:43.069 INFO:tasks.rados.rados.0.smithi170.stdout:202: left oid 14 (ObjNum 99 snap 13 seq_num 99) 2023-10-20T03:35:43.069 INFO:tasks.rados.rados.0.smithi170.stdout:207: finishing write tid 1 to smithi17039300-33 2023-10-20T03:35:43.069 INFO:tasks.rados.rados.0.smithi170.stdout:207: finishing write tid 2 to smithi17039300-33 2023-10-20T03:35:43.069 INFO:tasks.rados.rados.0.smithi170.stdout:207: finishing write tid 3 to smithi17039300-33 2023-10-20T03:35:43.069 INFO:tasks.rados.rados.0.smithi170.stdout:207: finishing write tid 4 to smithi17039300-33 2023-10-20T03:35:43.069 INFO:tasks.rados.rados.0.smithi170.stdout:207: finishing write tid 5 to smithi17039300-33 2023-10-20T03:35:43.070 INFO:tasks.rados.rados.0.smithi170.stdout:207: finishing write tid 6 to smithi17039300-33 2023-10-20T03:35:43.070 INFO:tasks.rados.rados.0.smithi170.stdout:207: oid 33 updating version 0 to 50 2023-10-20T03:35:43.070 INFO:tasks.rados.rados.0.smithi170.stdout:207: oid 33 updating version 50 to 51 2023-10-20T03:35:43.070 INFO:tasks.rados.rados.0.smithi170.stdout:207: oid 33 updating version 51 to 52 2023-10-20T03:35:43.070 INFO:tasks.rados.rados.0.smithi170.stdout:207: oid 33 version 52 is already newer than 48 2023-10-20T03:35:43.070 INFO:tasks.rados.rados.0.smithi170.stdout:207: oid 33 version 52 is already newer than 49 2023-10-20T03:35:43.071 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 33 v 52 (ObjNum 101 snap 13 seq_num 101) dirty exists 2023-10-20T03:35:43.071 INFO:tasks.rados.rados.0.smithi170.stdout:207: left oid 33 (ObjNum 101 snap 13 seq_num 101) 2023-10-20T03:35:43.071 INFO:tasks.rados.rados.0.smithi170.stdout:193: done (12 left) 2023-10-20T03:35:43.071 INFO:tasks.rados.rados.0.smithi170.stdout:195: done (11 left) 2023-10-20T03:35:43.071 INFO:tasks.rados.rados.0.smithi170.stdout:196: done (10 left) 2023-10-20T03:35:43.071 INFO:tasks.rados.rados.0.smithi170.stdout:197: done (9 left) 2023-10-20T03:35:43.072 INFO:tasks.rados.rados.0.smithi170.stdout:198: done (8 left) 2023-10-20T03:35:43.072 INFO:tasks.rados.rados.0.smithi170.stdout:199: done (7 left) 2023-10-20T03:35:43.072 INFO:tasks.rados.rados.0.smithi170.stdout:201: done (6 left) 2023-10-20T03:35:43.072 INFO:tasks.rados.rados.0.smithi170.stdout:202: done (5 left) 2023-10-20T03:35:43.072 INFO:tasks.rados.rados.0.smithi170.stdout:203: done (4 left) 2023-10-20T03:35:43.072 INFO:tasks.rados.rados.0.smithi170.stdout:205: done (3 left) 2023-10-20T03:35:43.073 INFO:tasks.rados.rados.0.smithi170.stdout:206: done (2 left) 2023-10-20T03:35:43.073 INFO:tasks.rados.rados.0.smithi170.stdout:207: done (1 left) 2023-10-20T03:35:43.073 INFO:tasks.rados.rados.0.smithi170.stdout:208: done (0 left) 2023-10-20T03:35:43.073 INFO:tasks.rados.rados.0.smithi170.stdout:209: rmattr oid 40 current snap is 13 2023-10-20T03:35:43.073 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 4 current snap is 13 2023-10-20T03:35:43.073 INFO:tasks.rados.rados.0.smithi170.stdout:210: seq_num 103 ranges {1933312=573440,2506752=786432,3293184=376832} 2023-10-20T03:35:43.074 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-10-20T03:35:43.078 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:43.076+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:43.097 INFO:tasks.rados.rados.0.smithi170.stdout:210: writing smithi17039300-4 from 1933312 to 2506752 tid 1 2023-10-20T03:35:43.108 INFO:tasks.rados.rados.0.smithi170.stdout:210: writing smithi17039300-4 from 2506752 to 3293184 tid 2 2023-10-20T03:35:43.112 INFO:tasks.rados.rados.0.smithi170.stdout:210: writing smithi17039300-4 from 3293184 to 3670016 tid 3 2023-10-20T03:35:43.113 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 32 current snap is 13 2023-10-20T03:35:43.113 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 23 (ObjNum 39 snap 0 seq_num 39) dirty exists 2023-10-20T03:35:43.113 INFO:tasks.rados.rados.0.smithi170.stdout:211: seq_num 104 ranges {1622016=589824,2211840=540672,2752512=622592} 2023-10-20T03:35:43.137 INFO:tasks.rados.rados.0.smithi170.stdout:211: writing smithi17039300-32 from 1622016 to 2211840 tid 1 2023-10-20T03:35:43.143 INFO:tasks.rados.rados.0.smithi170.stdout:211: writing smithi17039300-32 from 2211840 to 2752512 tid 2 2023-10-20T03:35:43.149 INFO:tasks.rados.rados.0.smithi170.stdout:211: writing smithi17039300-32 from 2752512 to 3375104 tid 3 2023-10-20T03:35:43.150 INFO:tasks.rados.rados.0.smithi170.stdout:209: done (2 left) 2023-10-20T03:35:43.150 INFO:tasks.rados.rados.0.smithi170.stdout:212: read oid 35 snap -1 2023-10-20T03:35:43.150 INFO:tasks.rados.rados.0.smithi170.stdout:212: expect (ObjNum 34 snap 0 seq_num 34) 2023-10-20T03:35:43.150 INFO:tasks.rados.rados.0.smithi170.stdout:213: read oid 14 snap -1 2023-10-20T03:35:43.151 INFO:tasks.rados.rados.0.smithi170.stdout:213: expect (ObjNum 99 snap 13 seq_num 99) 2023-10-20T03:35:43.151 INFO:tasks.rados.rados.0.smithi170.stdout:214: snap_create 2023-10-20T03:35:43.151 INFO:tasks.rados.rados.0.smithi170.stdout:210: finishing write tid 1 to smithi17039300-4 2023-10-20T03:35:43.152 INFO:tasks.rados.rados.0.smithi170.stdout:210: finishing write tid 2 to smithi17039300-4 2023-10-20T03:35:43.152 INFO:tasks.rados.rados.0.smithi170.stdout:210: finishing write tid 3 to smithi17039300-4 2023-10-20T03:35:43.154 INFO:tasks.rados.rados.0.smithi170.stdout:210: finishing write tid 4 to smithi17039300-4 2023-10-20T03:35:43.162 INFO:tasks.rados.rados.0.smithi170.stdout:210: finishing write tid 5 to smithi17039300-4 2023-10-20T03:35:43.162 INFO:tasks.rados.rados.0.smithi170.stdout:210: oid 4 updating version 0 to 49 2023-10-20T03:35:43.162 INFO:tasks.rados.rados.0.smithi170.stdout:210: oid 4 version 49 is already newer than 48 2023-10-20T03:35:43.162 INFO:tasks.rados.rados.0.smithi170.stdout:210: oid 4 updating version 49 to 50 2023-10-20T03:35:43.163 INFO:tasks.rados.rados.0.smithi170.stdout:210: oid 4 updating version 50 to 51 2023-10-20T03:35:43.163 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 51 (ObjNum 103 snap 13 seq_num 103) dirty exists 2023-10-20T03:35:43.163 INFO:tasks.rados.rados.0.smithi170.stdout:210: left oid 4 (ObjNum 103 snap 13 seq_num 103) 2023-10-20T03:35:43.164 INFO:tasks.rados.rados.0.smithi170.stdout:212: expect (ObjNum 34 snap 0 seq_num 34) 2023-10-20T03:35:43.178 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:43.174+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:43.195 INFO:tasks.rados.rados.0.smithi170.stdout:211: finishing write tid 1 to smithi17039300-32 2023-10-20T03:35:43.196 INFO:tasks.rados.rados.0.smithi170.stdout:211: finishing write tid 2 to smithi17039300-32 2023-10-20T03:35:43.196 INFO:tasks.rados.rados.0.smithi170.stdout:211: finishing write tid 3 to smithi17039300-32 2023-10-20T03:35:43.196 INFO:tasks.rados.rados.0.smithi170.stdout:211: finishing write tid 4 to smithi17039300-32 2023-10-20T03:35:43.196 INFO:tasks.rados.rados.0.smithi170.stdout:211: finishing write tid 5 to smithi17039300-32 2023-10-20T03:35:43.197 INFO:tasks.rados.rados.0.smithi170.stdout:211: oid 32 updating version 0 to 11 2023-10-20T03:35:43.197 INFO:tasks.rados.rados.0.smithi170.stdout:211: oid 32 version 11 is already newer than 10 2023-10-20T03:35:43.197 INFO:tasks.rados.rados.0.smithi170.stdout:211: oid 32 updating version 11 to 12 2023-10-20T03:35:43.197 INFO:tasks.rados.rados.0.smithi170.stdout:211: oid 32 version 12 is already newer than 9 2023-10-20T03:35:43.197 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 12 (ObjNum 104 snap 13 seq_num 104) dirty exists 2023-10-20T03:35:43.198 INFO:tasks.rados.rados.0.smithi170.stdout:211: left oid 32 (ObjNum 104 snap 13 seq_num 104) 2023-10-20T03:35:43.202 INFO:tasks.rados.rados.0.smithi170.stdout:213: expect (ObjNum 99 snap 13 seq_num 99) 2023-10-20T03:35:43.209 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:43.209 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:43.223 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-10-20T03:35:43.279 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:43.276+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:43.349 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:43.349 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:43.363 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-10-20T03:35:43.410 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:43.409+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:43.481 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:43.478+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:43.490 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:43.491 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:43.505 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-10-20T03:35:43.582 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:43.580+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:43.677 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:43.677 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:43.682 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:43.681+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:43.692 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-10-20T03:35:43.783 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:43.782+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:43.849 INFO:tasks.rados.rados.0.smithi170.stdout:210: done (4 left) 2023-10-20T03:35:43.850 INFO:tasks.rados.rados.0.smithi170.stdout:211: done (3 left) 2023-10-20T03:35:43.850 INFO:tasks.rados.rados.0.smithi170.stdout:212: done (2 left) 2023-10-20T03:35:43.850 INFO:tasks.rados.rados.0.smithi170.stdout:213: done (1 left) 2023-10-20T03:35:43.851 INFO:tasks.rados.rados.0.smithi170.stdout:214: done (0 left) 2023-10-20T03:35:43.851 INFO:tasks.rados.rados.0.smithi170.stdout:215: read oid 44 snap -1 2023-10-20T03:35:43.851 INFO:tasks.rados.rados.0.smithi170.stdout:215: expect (ObjNum 43 snap 0 seq_num 43) 2023-10-20T03:35:43.851 INFO:tasks.rados.rados.0.smithi170.stdout:216: rollback oid 8 current snap is 14 2023-10-20T03:35:43.851 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 8 to 13 2023-10-20T03:35:43.852 INFO:tasks.rados.rados.0.smithi170.stdout:217: snap_create 2023-10-20T03:35:43.871 INFO:tasks.rados.rados.0.smithi170.stdout:215: expect (ObjNum 43 snap 0 seq_num 43) 2023-10-20T03:35:43.872 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:43.872 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:43.883 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:43.882+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:43.886 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-10-20T03:35:43.891 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-10-20T03:35:43.892 INFO:tasks.rados.rados.0.smithi170.stdout:216: finishing rollback tid 0 to smithi17039300-8 2023-10-20T03:35:43.892 INFO:tasks.rados.rados.0.smithi170.stdout:216: finishing rollback tid 1 to smithi17039300-8 2023-10-20T03:35:43.892 INFO:tasks.rados.rados.0.smithi170.stdout:216: finishing rollback tid 2 to smithi17039300-8 2023-10-20T03:35:43.892 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 8 v 58 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-10-20T03:35:43.985 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:43.981+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:44.014 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:44.014 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:44.029 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-10-20T03:35:44.122 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:44.121+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:44.187 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:44.184+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:44.190 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:44.191 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:44.205 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-10-20T03:35:44.288 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:44.284+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:44.371 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:44.372 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:44.386 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-10-20T03:35:44.388 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:44.384+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:44.489 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:44.484+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:44.514 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:44.514 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:44.528 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-10-20T03:35:44.590 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:44.588+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:44.662 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:44.662 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:44.678 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-10-20T03:35:44.691 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:44.690+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:44.791 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:44.788+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:44.843 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:44.843 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:44.857 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-10-20T03:35:44.861 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2023-10-20T03:35:44.865 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:35:44.868 INFO:tasks.rados.rados.0.smithi170.stdout:215: done (2 left) 2023-10-20T03:35:44.868 INFO:tasks.rados.rados.0.smithi170.stdout:216: done (1 left) 2023-10-20T03:35:44.868 INFO:tasks.rados.rados.0.smithi170.stdout:217: done (0 left) 2023-10-20T03:35:44.869 INFO:tasks.rados.rados.0.smithi170.stdout:218: read oid 29 snap -1 2023-10-20T03:35:44.869 INFO:tasks.rados.rados.0.smithi170.stdout:218: expect deleted 2023-10-20T03:35:44.869 INFO:tasks.rados.rados.0.smithi170.stdout:219: rmattr oid 23 current snap is 15 2023-10-20T03:35:44.869 INFO:tasks.rados.rados.0.smithi170.stdout:219: done (1 left) 2023-10-20T03:35:44.869 INFO:tasks.rados.rados.0.smithi170.stdout:220: rmattr oid 42 current snap is 15 2023-10-20T03:35:44.870 INFO:tasks.rados.rados.0.smithi170.stdout:221: snap_create 2023-10-20T03:35:44.883 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 42 v 37 (ObjNum 60 snap 2 seq_num 60) dirty exists 2023-10-20T03:35:44.899 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:44.896+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:44.993 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:44.990+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:45.010 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:45.010 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.025 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:45.025 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:45.030 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:45.031 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.039 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-10-20T03:35:45.050 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:45.051 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.071 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:45.071 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.088 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:45.089 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.093 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:45.089+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:45.106 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.107 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:45.126 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:45.127 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.144 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:45.144 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.164 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:45.165 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.166 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:45.166 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:45.166 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:35:45.166 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:45.181 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-10-20T03:35:45.185 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:45.186 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.208 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:45.208 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.227 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:45.228 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.230 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:45.228+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:45.248 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:45.248 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.271 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:45.272 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.295 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:45.295 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:35:45.296 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:45.292+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:45.301 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:45.301 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:45.307 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:45.307 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:45.308 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:35:45.308 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:35:45.308 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:45.321 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-10-20T03:35:45.396 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:45.396+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:45.493 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:45.493 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:35:45.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:35:45.494 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2023-10-20T03:35:45.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.494 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=23, reqid=client.4570.0:208, at_version=37'14, trim_to=0'0, roll_forward_to=36'12))", 2023-10-20T03:35:45.494 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:19.272177+0000", 2023-10-20T03:35:45.495 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.217669000000001, 2023-10-20T03:35:45.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010004097, 2023-10-20T03:35:45.495 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.495 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.495 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.496 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.496 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272177+0000", 2023-10-20T03:35:45.496 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272177+0000", 2023-10-20T03:35:45.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.498 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.498 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272177+0000", 2023-10-20T03:35:45.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5800000000000001e-07 2023-10-20T03:35:45.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.500 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272691+0000", 2023-10-20T03:35:45.500 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051351599999999999 2023-10-20T03:35:45.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.501 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272709+0000", 2023-10-20T03:35:45.501 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8108999999999999e-05 2023-10-20T03:35:45.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272738+0000", 2023-10-20T03:35:45.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9246000000000001e-05 2023-10-20T03:35:45.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272855+0000", 2023-10-20T03:35:45.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011678 2023-10-20T03:35:45.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272869+0000", 2023-10-20T03:35:45.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4409999999999999e-05 2023-10-20T03:35:45.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.505 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.505 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282065+0000", 2023-10-20T03:35:45.505 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0091960219999999999 2023-10-20T03:35:45.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.506 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.506 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.506 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282181+0000", 2023-10-20T03:35:45.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011575599999999999 2023-10-20T03:35:45.506 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.507 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.507 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.507 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=24, reqid=client.4570.0:209, at_version=37'15, trim_to=0'0, roll_forward_to=36'12))", 2023-10-20T03:35:45.508 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:19.288930+0000", 2023-10-20T03:35:45.508 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.200916173, 2023-10-20T03:35:45.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049673369999999996, 2023-10-20T03:35:45.508 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.508 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.508 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.509 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.288930+0000", 2023-10-20T03:35:45.509 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.288930+0000", 2023-10-20T03:35:45.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.288932+0000", 2023-10-20T03:35:45.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1789999999999998e-06 2023-10-20T03:35:45.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.513 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.513 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.289148+0000", 2023-10-20T03:35:45.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000215465 2023-10-20T03:35:45.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.289179+0000", 2023-10-20T03:35:45.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1744e-05 2023-10-20T03:35:45.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.289203+0000", 2023-10-20T03:35:45.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3932999999999999e-05 2023-10-20T03:35:45.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.289297+0000", 2023-10-20T03:35:45.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4096000000000002e-05 2023-10-20T03:35:45.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.289307+0000", 2023-10-20T03:35:45.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5040000000000008e-06 2023-10-20T03:35:45.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.517 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.293842+0000", 2023-10-20T03:35:45.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045355079999999997 2023-10-20T03:35:45.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.518 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.293897+0000", 2023-10-20T03:35:45.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4907999999999997e-05 2023-10-20T03:35:45.518 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.519 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.519 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.519 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 43/31 ECSubWrite(tid=18, reqid=client.4570.0:316, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2023-10-20T03:35:45.519 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.368510+0000", 2023-10-20T03:35:45.519 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.121336141, 2023-10-20T03:35:45.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010086807, 2023-10-20T03:35:45.520 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.520 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.520 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368510+0000", 2023-10-20T03:35:45.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.522 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368510+0000", 2023-10-20T03:35:45.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368510+0000", 2023-10-20T03:35:45.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7999999999999995e-07 2023-10-20T03:35:45.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368574+0000", 2023-10-20T03:35:45.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3555999999999999e-05 2023-10-20T03:35:45.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368612+0000", 2023-10-20T03:35:45.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8386999999999997e-05 2023-10-20T03:35:45.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368638+0000", 2023-10-20T03:35:45.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5775999999999999e-05 2023-10-20T03:35:45.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368887+0000", 2023-10-20T03:35:45.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024854500000000003 2023-10-20T03:35:45.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368910+0000", 2023-10-20T03:35:45.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.353e-05 2023-10-20T03:35:45.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.378493+0000", 2023-10-20T03:35:45.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0095828070000000005 2023-10-20T03:35:45.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.378597+0000", 2023-10-20T03:35:45.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000103626 2023-10-20T03:35:45.532 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.532 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.533 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.533 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 43/31 ECSubWrite(tid=19, reqid=client.4570.0:317, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2023-10-20T03:35:45.534 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.369969+0000", 2023-10-20T03:35:45.534 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.119877368000001, 2023-10-20T03:35:45.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010764182000000001, 2023-10-20T03:35:45.535 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.535 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.535 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369969+0000", 2023-10-20T03:35:45.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369969+0000", 2023-10-20T03:35:45.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369971+0000", 2023-10-20T03:35:45.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8360000000000001e-06 2023-10-20T03:35:45.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370018+0000", 2023-10-20T03:35:45.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7976999999999998e-05 2023-10-20T03:35:45.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370050+0000", 2023-10-20T03:35:45.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1668999999999998e-05 2023-10-20T03:35:45.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370078+0000", 2023-10-20T03:35:45.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8019999999999999e-05 2023-10-20T03:35:45.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.542 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.371731+0000", 2023-10-20T03:35:45.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0016532039999999999 2023-10-20T03:35:45.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.543 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.543 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.371760+0000", 2023-10-20T03:35:45.543 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.904e-05 2023-10-20T03:35:45.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.543 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.544 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.380637+0000", 2023-10-20T03:35:45.544 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0088761570000000008 2023-10-20T03:35:45.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.544 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.380733+0000", 2023-10-20T03:35:45.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6279000000000004e-05 2023-10-20T03:35:45.545 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.545 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.545 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.546 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 43/31 ECSubWrite(tid=20, reqid=client.4570.0:318, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2023-10-20T03:35:45.546 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.370315+0000", 2023-10-20T03:35:45.546 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.119531411000001, 2023-10-20T03:35:45.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010484894, 2023-10-20T03:35:45.547 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.547 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.547 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.548 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370315+0000", 2023-10-20T03:35:45.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.548 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370315+0000", 2023-10-20T03:35:45.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370316+0000", 2023-10-20T03:35:45.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4699999999999999e-06 2023-10-20T03:35:45.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370356+0000", 2023-10-20T03:35:45.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9996000000000001e-05 2023-10-20T03:35:45.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370382+0000", 2023-10-20T03:35:45.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6310000000000001e-05 2023-10-20T03:35:45.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370403+0000", 2023-10-20T03:35:45.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0784e-05 2023-10-20T03:35:45.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.553 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.374885+0000", 2023-10-20T03:35:45.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0044819090000000001 2023-10-20T03:35:45.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.374981+0000", 2023-10-20T03:35:45.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5618999999999994e-05 2023-10-20T03:35:45.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.380741+0000", 2023-10-20T03:35:45.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057606469999999998 2023-10-20T03:35:45.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.380800+0000", 2023-10-20T03:35:45.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8158999999999997e-05 2023-10-20T03:35:45.556 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.557 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.557 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.557 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:45.558 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.387934+0000", 2023-10-20T03:35:45.558 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 19.101912428999999, 2023-10-20T03:35:45.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033593076999999999, 2023-10-20T03:35:45.558 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.558 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.559 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387934+0000", 2023-10-20T03:35:45.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387934+0000", 2023-10-20T03:35:45.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387934+0000", 2023-10-20T03:35:45.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7e-07 2023-10-20T03:35:45.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388584+0000", 2023-10-20T03:35:45.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00064982500000000003 2023-10-20T03:35:45.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388724+0000", 2023-10-20T03:35:45.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013997200000000001 2023-10-20T03:35:45.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388764+0000", 2023-10-20T03:35:45.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0352999999999998e-05 2023-10-20T03:35:45.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388939+0000", 2023-10-20T03:35:45.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017499200000000001 2023-10-20T03:35:45.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388956+0000", 2023-10-20T03:35:45.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6990000000000002e-05 2023-10-20T03:35:45.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421453+0000", 2023-10-20T03:35:45.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032496615999999999 2023-10-20T03:35:45.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421527+0000", 2023-10-20T03:35:45.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3959000000000001e-05 2023-10-20T03:35:45.568 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.568 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.568 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.569 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/31 ECSubWrite(tid=22, reqid=client.4570.0:340, at_version=44'12, trim_to=0'0, roll_forward_to=43'10))", 2023-10-20T03:35:45.569 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:35:45.569 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 19.090936609, 2023-10-20T03:35:45.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022077208000000001, 2023-10-20T03:35:45.570 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.570 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.570 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:35:45.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:35:45.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398910+0000", 2023-10-20T03:35:45.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6400000000000002e-07 2023-10-20T03:35:45.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.574 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399138+0000", 2023-10-20T03:35:45.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022810800000000001 2023-10-20T03:35:45.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.575 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.575 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399169+0000", 2023-10-20T03:35:45.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0593000000000003e-05 2023-10-20T03:35:45.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399190+0000", 2023-10-20T03:35:45.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.12e-05 2023-10-20T03:35:45.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399353+0000", 2023-10-20T03:35:45.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016285200000000001 2023-10-20T03:35:45.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.578 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399378+0000", 2023-10-20T03:35:45.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5077000000000001e-05 2023-10-20T03:35:45.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.579 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420889+0000", 2023-10-20T03:35:45.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021511188000000001 2023-10-20T03:35:45.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420987+0000", 2023-10-20T03:35:45.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7626000000000003e-05 2023-10-20T03:35:45.581 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.581 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.581 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.582 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:45.582 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.401022+0000", 2023-10-20T03:35:45.582 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 19.088823574999999, 2023-10-20T03:35:45.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020571461999999999, 2023-10-20T03:35:45.583 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.583 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.583 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.583 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.583 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401022+0000", 2023-10-20T03:35:45.584 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.584 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.584 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401022+0000", 2023-10-20T03:35:45.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401024+0000", 2023-10-20T03:35:45.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.877e-06 2023-10-20T03:35:45.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401473+0000", 2023-10-20T03:35:45.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044877699999999998 2023-10-20T03:35:45.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401507+0000", 2023-10-20T03:35:45.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4209999999999999e-05 2023-10-20T03:35:45.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401543+0000", 2023-10-20T03:35:45.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6003000000000001e-05 2023-10-20T03:35:45.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401767+0000", 2023-10-20T03:35:45.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022351200000000001 2023-10-20T03:35:45.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401793+0000", 2023-10-20T03:35:45.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6529e-05 2023-10-20T03:35:45.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421560+0000", 2023-10-20T03:35:45.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019766305000000001 2023-10-20T03:35:45.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421594+0000", 2023-10-20T03:35:45.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4248999999999997e-05 2023-10-20T03:35:45.592 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.593 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.593 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.593 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 44/31 ECSubWrite(tid=53, reqid=client.4570.0:335, at_version=44'20, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:45.594 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.402960+0000", 2023-10-20T03:35:45.594 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 19.086885542000001, 2023-10-20T03:35:45.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019889799999999999, 2023-10-20T03:35:45.594 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.594 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.595 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402960+0000", 2023-10-20T03:35:45.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402960+0000", 2023-10-20T03:35:45.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402962+0000", 2023-10-20T03:35:45.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8470000000000001e-06 2023-10-20T03:35:45.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403065+0000", 2023-10-20T03:35:45.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000102179 2023-10-20T03:35:45.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403099+0000", 2023-10-20T03:35:45.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4813e-05 2023-10-20T03:35:45.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403122+0000", 2023-10-20T03:35:45.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2226e-05 2023-10-20T03:35:45.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419074+0000", 2023-10-20T03:35:45.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015952643999999998 2023-10-20T03:35:45.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419093+0000", 2023-10-20T03:35:45.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9089e-05 2023-10-20T03:35:45.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.422750+0000", 2023-10-20T03:35:45.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0036562959999999999 2023-10-20T03:35:45.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.422850+0000", 2023-10-20T03:35:45.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010070599999999999 2023-10-20T03:35:45.605 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.605 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.606 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.607 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 44/31 ECSubWrite(tid=54, reqid=client.4570.0:336, at_version=44'21, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:45.607 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.404639+0000", 2023-10-20T03:35:45.608 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 19.085206941999999, 2023-10-20T03:35:45.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018626967000000001, 2023-10-20T03:35:45.608 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.609 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.609 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404639+0000", 2023-10-20T03:35:45.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404639+0000", 2023-10-20T03:35:45.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404641+0000", 2023-10-20T03:35:45.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7969999999999999e-06 2023-10-20T03:35:45.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404707+0000", 2023-10-20T03:35:45.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6523e-05 2023-10-20T03:35:45.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404741+0000", 2023-10-20T03:35:45.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4078999999999997e-05 2023-10-20T03:35:45.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404764+0000", 2023-10-20T03:35:45.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.209e-05 2023-10-20T03:35:45.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.616 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420947+0000", 2023-10-20T03:35:45.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016183831999999999 2023-10-20T03:35:45.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420971+0000", 2023-10-20T03:35:45.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.332e-05 2023-10-20T03:35:45.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.423172+0000", 2023-10-20T03:35:45.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002200867 2023-10-20T03:35:45.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.423266+0000", 2023-10-20T03:35:45.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4458999999999998e-05 2023-10-20T03:35:45.619 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.619 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.619 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.620 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:35:45.620 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:30.392373+0000", 2023-10-20T03:35:45.620 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 15.097472731, 2023-10-20T03:35:45.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038515463, 2023-10-20T03:35:45.621 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.621 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.621 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.392373+0000", 2023-10-20T03:35:45.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.622 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.392373+0000", 2023-10-20T03:35:45.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.392374+0000", 2023-10-20T03:35:45.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3e-07 2023-10-20T03:35:45.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394109+0000", 2023-10-20T03:35:45.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001734977 2023-10-20T03:35:45.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394151+0000", 2023-10-20T03:35:45.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1872999999999997e-05 2023-10-20T03:35:45.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394177+0000", 2023-10-20T03:35:45.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5636e-05 2023-10-20T03:35:45.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394320+0000", 2023-10-20T03:35:45.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014345499999999999 2023-10-20T03:35:45.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394348+0000", 2023-10-20T03:35:45.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7523999999999999e-05 2023-10-20T03:35:45.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.430793+0000", 2023-10-20T03:35:45.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.036445549000000001 2023-10-20T03:35:45.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.630 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.630 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.430889+0000", 2023-10-20T03:35:45.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5718999999999996e-05 2023-10-20T03:35:45.631 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.631 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.631 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.631 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 49/31 ECSubWrite(tid=24, reqid=client.4570.0:392, at_version=49'14, trim_to=0'0, roll_forward_to=44'12))", 2023-10-20T03:35:45.632 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.388889+0000", 2023-10-20T03:35:45.632 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 14.100957371, 2023-10-20T03:35:45.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0086355749999999995, 2023-10-20T03:35:45.632 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.632 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.632 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388889+0000", 2023-10-20T03:35:45.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.634 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.634 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388889+0000", 2023-10-20T03:35:45.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.635 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388889+0000", 2023-10-20T03:35:45.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.13e-07 2023-10-20T03:35:45.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388970+0000", 2023-10-20T03:35:45.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1123000000000002e-05 2023-10-20T03:35:45.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.637 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389004+0000", 2023-10-20T03:35:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3946000000000003e-05 2023-10-20T03:35:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389029+0000", 2023-10-20T03:35:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4187000000000001e-05 2023-10-20T03:35:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389281+0000", 2023-10-20T03:35:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025258799999999999 2023-10-20T03:35:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389311+0000", 2023-10-20T03:35:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.993e-05 2023-10-20T03:35:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.397420+0000", 2023-10-20T03:35:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0081085359999999995 2023-10-20T03:35:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.397524+0000", 2023-10-20T03:35:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104652 2023-10-20T03:35:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 49/31 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=49'16, trim_to=0'0, roll_forward_to=44'12))", 2023-10-20T03:35:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.392300+0000", 2023-10-20T03:35:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 14.09754592, 2023-10-20T03:35:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0065106510000000001, 2023-10-20T03:35:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.392300+0000", 2023-10-20T03:35:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.392300+0000", 2023-10-20T03:35:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.392302+0000", 2023-10-20T03:35:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.996e-06 2023-10-20T03:35:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.392360+0000", 2023-10-20T03:35:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7803000000000002e-05 2023-10-20T03:35:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.392393+0000", 2023-10-20T03:35:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3442999999999998e-05 2023-10-20T03:35:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.392417+0000", 2023-10-20T03:35:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3697e-05 2023-10-20T03:35:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.392590+0000", 2023-10-20T03:35:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017258200000000001 2023-10-20T03:35:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.392618+0000", 2023-10-20T03:35:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8430000000000001e-05 2023-10-20T03:35:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.398721+0000", 2023-10-20T03:35:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0061026969999999998 2023-10-20T03:35:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.398811+0000", 2023-10-20T03:35:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0002999999999999e-05 2023-10-20T03:35:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 49/31 ECSubWrite(tid=63, reqid=osd.12.0:63, at_version=49'27, trim_to=0'0, roll_forward_to=48'23))", 2023-10-20T03:35:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.408201+0000", 2023-10-20T03:35:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 14.081645036999999, 2023-10-20T03:35:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0056145279999999997, 2023-10-20T03:35:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.408201+0000", 2023-10-20T03:35:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.408201+0000", 2023-10-20T03:35:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.408202+0000", 2023-10-20T03:35:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0070000000000001e-06 2023-10-20T03:35:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.408275+0000", 2023-10-20T03:35:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2595999999999994e-05 2023-10-20T03:35:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.408303+0000", 2023-10-20T03:35:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8269999999999999e-05 2023-10-20T03:35:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.408322+0000", 2023-10-20T03:35:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.948e-05 2023-10-20T03:35:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.409712+0000", 2023-10-20T03:35:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0013893390000000001 2023-10-20T03:35:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.409740+0000", 2023-10-20T03:35:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8629999999999999e-05 2023-10-20T03:35:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.413721+0000", 2023-10-20T03:35:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039809399999999997 2023-10-20T03:35:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.413816+0000", 2023-10-20T03:35:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4265999999999995e-05 2023-10-20T03:35:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 53/31 ECSubWrite(tid=33, reqid=osd.8.0:33, at_version=53'24, trim_to=0'0, roll_forward_to=52'22))", 2023-10-20T03:35:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:35.431073+0000", 2023-10-20T03:35:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.058772962999999, 2023-10-20T03:35:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0076161019999999996, 2023-10-20T03:35:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431073+0000", 2023-10-20T03:35:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431073+0000", 2023-10-20T03:35:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431074+0000", 2023-10-20T03:35:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4999999999999998e-07 2023-10-20T03:35:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431152+0000", 2023-10-20T03:35:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8329000000000002e-05 2023-10-20T03:35:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431270+0000", 2023-10-20T03:35:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000118403 2023-10-20T03:35:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431298+0000", 2023-10-20T03:35:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7512999999999999e-05 2023-10-20T03:35:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431585+0000", 2023-10-20T03:35:45.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028691399999999998 2023-10-20T03:35:45.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431614+0000", 2023-10-20T03:35:45.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9280000000000001e-05 2023-10-20T03:35:45.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.438579+0000", 2023-10-20T03:35:45.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0069652580000000002 2023-10-20T03:35:45.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.438689+0000", 2023-10-20T03:35:45.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109955 2023-10-20T03:35:45.682 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.682 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.683 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.683 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 60/56 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=60'45, trim_to=0'0, roll_forward_to=57'43))", 2023-10-20T03:35:45.683 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.830961+0000", 2023-10-20T03:35:45.684 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.6588849789999998, 2023-10-20T03:35:45.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090787750000000007, 2023-10-20T03:35:45.684 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.684 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.684 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.830961+0000", 2023-10-20T03:35:45.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.830961+0000", 2023-10-20T03:35:45.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.830962+0000", 2023-10-20T03:35:45.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.63e-07 2023-10-20T03:35:45.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.831005+0000", 2023-10-20T03:35:45.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3516000000000001e-05 2023-10-20T03:35:45.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.831037+0000", 2023-10-20T03:35:45.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2153999999999997e-05 2023-10-20T03:35:45.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.831066+0000", 2023-10-20T03:35:45.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9043e-05 2023-10-20T03:35:45.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.831253+0000", 2023-10-20T03:35:45.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000186711 2023-10-20T03:35:45.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.831286+0000", 2023-10-20T03:35:45.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3396999999999999e-05 2023-10-20T03:35:45.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.839932+0000", 2023-10-20T03:35:45.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0086460649999999997 2023-10-20T03:35:45.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.840040+0000", 2023-10-20T03:35:45.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107426 2023-10-20T03:35:45.694 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.694 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.694 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.695 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 60/56 ECSubWrite(tid=28, reqid=osd.1.0:28, at_version=60'47, trim_to=0'0, roll_forward_to=57'43))", 2023-10-20T03:35:45.695 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.834067+0000", 2023-10-20T03:35:45.695 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.6557794559999999, 2023-10-20T03:35:45.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010530794, 2023-10-20T03:35:45.697 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.697 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.697 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.834067+0000", 2023-10-20T03:35:45.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.834067+0000", 2023-10-20T03:35:45.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.834069+0000", 2023-10-20T03:35:45.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0770000000000002e-06 2023-10-20T03:35:45.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.834123+0000", 2023-10-20T03:35:45.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4611999999999998e-05 2023-10-20T03:35:45.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.834166+0000", 2023-10-20T03:35:45.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2370000000000003e-05 2023-10-20T03:35:45.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.834190+0000", 2023-10-20T03:35:45.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4573e-05 2023-10-20T03:35:45.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.836006+0000", 2023-10-20T03:35:45.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00181566 2023-10-20T03:35:45.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.836033+0000", 2023-10-20T03:35:45.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6973000000000001e-05 2023-10-20T03:35:45.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.844509+0000", 2023-10-20T03:35:45.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084763700000000004 2023-10-20T03:35:45.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.844597+0000", 2023-10-20T03:35:45.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8158999999999994e-05 2023-10-20T03:35:45.707 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.707 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.708 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.708 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 60/31 ECSubWrite(tid=39, reqid=osd.8.0:39, at_version=60'30, trim_to=0'0, roll_forward_to=56'28))", 2023-10-20T03:35:45.708 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.843614+0000", 2023-10-20T03:35:45.708 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.6462320940000001, 2023-10-20T03:35:45.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0074417670000000002, 2023-10-20T03:35:45.709 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.709 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.709 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843614+0000", 2023-10-20T03:35:45.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843614+0000", 2023-10-20T03:35:45.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843615+0000", 2023-10-20T03:35:45.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7999999999999995e-07 2023-10-20T03:35:45.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843670+0000", 2023-10-20T03:35:45.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5442999999999998e-05 2023-10-20T03:35:45.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843701+0000", 2023-10-20T03:35:45.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0985999999999998e-05 2023-10-20T03:35:45.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843727+0000", 2023-10-20T03:35:45.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5497e-05 2023-10-20T03:35:45.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.844678+0000", 2023-10-20T03:35:45.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00095127600000000003 2023-10-20T03:35:45.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.844706+0000", 2023-10-20T03:35:45.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8002999999999999e-05 2023-10-20T03:35:45.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.850934+0000", 2023-10-20T03:35:45.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062281630000000001 2023-10-20T03:35:45.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.851056+0000", 2023-10-20T03:35:45.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000121719 2023-10-20T03:35:45.719 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.719 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.719 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.720 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 60/56 ECSubWrite(tid=29, reqid=osd.1.0:28, at_version=0'0, trim_to=0'0, roll_forward_to=60'47))", 2023-10-20T03:35:45.720 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.847024+0000", 2023-10-20T03:35:45.720 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.6428223129999999, 2023-10-20T03:35:45.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0066434390000000001, 2023-10-20T03:35:45.720 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.721 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.721 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847024+0000", 2023-10-20T03:35:45.722 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.722 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.723 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847024+0000", 2023-10-20T03:35:45.723 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.723 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.723 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847026+0000", 2023-10-20T03:35:45.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0099999999999998e-06 2023-10-20T03:35:45.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847066+0000", 2023-10-20T03:35:45.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0111999999999998e-05 2023-10-20T03:35:45.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847100+0000", 2023-10-20T03:35:45.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4397000000000003e-05 2023-10-20T03:35:45.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847148+0000", 2023-10-20T03:35:45.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.816e-05 2023-10-20T03:35:45.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.727 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.727 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847316+0000", 2023-10-20T03:35:45.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016719799999999999 2023-10-20T03:35:45.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847345+0000", 2023-10-20T03:35:45.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.898e-05 2023-10-20T03:35:45.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.853568+0000", 2023-10-20T03:35:45.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.006223096 2023-10-20T03:35:45.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.853667+0000", 2023-10-20T03:35:45.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9486000000000003e-05 2023-10-20T03:35:45.731 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.731 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.731 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.732 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.cs2 60/56 ECSubWrite(tid=193, reqid=osd.13.0:193, at_version=60'54, trim_to=0'0, roll_forward_to=57'50))", 2023-10-20T03:35:45.732 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.866142+0000", 2023-10-20T03:35:45.732 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.6237044379999999, 2023-10-20T03:35:45.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0059042260000000003, 2023-10-20T03:35:45.732 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:45.733 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:45.733 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:45.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:45.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.866142+0000", 2023-10-20T03:35:45.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:45.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.866142+0000", 2023-10-20T03:35:45.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:45.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:45.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.866143+0000", 2023-10-20T03:35:45.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.47e-07 2023-10-20T03:35:45.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:45.737 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.866188+0000", 2023-10-20T03:35:45.737 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5819000000000003e-05 2023-10-20T03:35:45.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.737 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.737 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:45.738 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.866216+0000", 2023-10-20T03:35:45.738 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7137000000000001e-05 2023-10-20T03:35:45.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.738 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:45.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.866235+0000", 2023-10-20T03:35:45.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9783e-05 2023-10-20T03:35:45.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:45.740 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.867660+0000", 2023-10-20T03:35:45.740 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001424519 2023-10-20T03:35:45.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.740 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:45.740 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.867690+0000", 2023-10-20T03:35:45.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0369999999999999e-05 2023-10-20T03:35:45.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:45.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.871963+0000", 2023-10-20T03:35:45.742 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0042728990000000001 2023-10-20T03:35:45.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:45.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:45.742 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:45.743 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.872046+0000", 2023-10-20T03:35:45.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2751999999999996e-05 2023-10-20T03:35:45.743 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.743 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.744 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.744 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:45.744 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:45.744 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:45.745 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15] out_osds: [12] dead_osds: [6] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 10, 14, 3, 7, 11, 15] 2023-10-20T03:35:45.745 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-10-20T03:35:45.746 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-10-20T03:35:45.746 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.958457 2023-10-20T03:35:45.746 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0.958457121139527 2023-10-20T03:35:45.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-10-20T03:35:45.749 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:45.740+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:45.847 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:45.845+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:45.873 INFO:tasks.rados.rados.0.smithi170.stdout:218: done (2 left) 2023-10-20T03:35:45.873 INFO:tasks.rados.rados.0.smithi170.stdout:220: done (1 left) 2023-10-20T03:35:45.873 INFO:tasks.rados.rados.0.smithi170.stdout:221: done (0 left) 2023-10-20T03:35:45.874 INFO:tasks.rados.rados.0.smithi170.stdout:222: read oid 18 snap -1 2023-10-20T03:35:45.874 INFO:tasks.rados.rados.0.smithi170.stdout:222: expect deleted 2023-10-20T03:35:45.874 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 9 current snap is 16 2023-10-20T03:35:45.874 INFO:tasks.rados.rados.0.smithi170.stdout:223: seq_num 105 ranges {147456=606208,753664=688128,1441792=475136} 2023-10-20T03:35:45.884 INFO:tasks.rados.rados.0.smithi170.stdout:223: writing smithi17039300-9 from 147456 to 753664 tid 1 2023-10-20T03:35:45.895 INFO:tasks.rados.rados.0.smithi170.stdout:223: writing smithi17039300-9 from 753664 to 1441792 tid 2 2023-10-20T03:35:45.903 INFO:tasks.rados.rados.0.smithi170.stdout:223: writing smithi17039300-9 from 1441792 to 1916928 tid 3 2023-10-20T03:35:45.903 INFO:tasks.rados.rados.0.smithi170.stdout:224: snap_remove snap 13 2023-10-20T03:35:45.930 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:35:45.930 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:45.947 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-10-20T03:35:45.948 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:45.946+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:46.048 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:46.048+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:46.073 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:46.074 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:46.074 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:35:46.074 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:35:46.074 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:46.088 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-10-20T03:35:46.149 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:46.144+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:46.217 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:46.218 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:35:46.218 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:35:46.218 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2023-10-20T03:35:46.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.218 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s4 43/31 ECSubWrite(tid=18, reqid=client.4570.0:316, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2023-10-20T03:35:46.219 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.368494+0000", 2023-10-20T03:35:46.219 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.845657397, 2023-10-20T03:35:46.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0086458249999999993, 2023-10-20T03:35:46.219 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.219 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.220 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.220 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.220 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368494+0000", 2023-10-20T03:35:46.220 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.221 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.221 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368494+0000", 2023-10-20T03:35:46.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368495+0000", 2023-10-20T03:35:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4700000000000002e-07 2023-10-20T03:35:46.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368554+0000", 2023-10-20T03:35:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9540000000000003e-05 2023-10-20T03:35:46.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368609+0000", 2023-10-20T03:35:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5002e-05 2023-10-20T03:35:46.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368635+0000", 2023-10-20T03:35:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5372999999999999e-05 2023-10-20T03:35:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368932+0000", 2023-10-20T03:35:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000297585 2023-10-20T03:35:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368961+0000", 2023-10-20T03:35:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8413000000000001e-05 2023-10-20T03:35:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.377036+0000", 2023-10-20T03:35:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.008074945 2023-10-20T03:35:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.377140+0000", 2023-10-20T03:35:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010462 2023-10-20T03:35:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s4 43/31 ECSubWrite(tid=19, reqid=client.4570.0:317, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2023-10-20T03:35:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.369790+0000", 2023-10-20T03:35:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.844361376999998, 2023-10-20T03:35:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011264842000000001, 2023-10-20T03:35:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369790+0000", 2023-10-20T03:35:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369790+0000", 2023-10-20T03:35:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369793+0000", 2023-10-20T03:35:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1270000000000001e-06 2023-10-20T03:35:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369842+0000", 2023-10-20T03:35:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9456000000000002e-05 2023-10-20T03:35:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369906+0000", 2023-10-20T03:35:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4313000000000006e-05 2023-10-20T03:35:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369931+0000", 2023-10-20T03:35:46.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4695999999999999e-05 2023-10-20T03:35:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.371584+0000", 2023-10-20T03:35:46.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0016525450000000001 2023-10-20T03:35:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.371614+0000", 2023-10-20T03:35:46.238 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0496000000000001e-05 2023-10-20T03:35:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.380965+0000", 2023-10-20T03:35:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0093513999999999993 2023-10-20T03:35:46.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.381055+0000", 2023-10-20T03:35:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9808999999999993e-05 2023-10-20T03:35:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.240 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s4 43/31 ECSubWrite(tid=20, reqid=client.4570.0:318, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2023-10-20T03:35:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.370556+0000", 2023-10-20T03:35:46.241 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.843595559000001, 2023-10-20T03:35:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010561004000000001, 2023-10-20T03:35:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370556+0000", 2023-10-20T03:35:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370556+0000", 2023-10-20T03:35:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370558+0000", 2023-10-20T03:35:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7129999999999999e-06 2023-10-20T03:35:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370603+0000", 2023-10-20T03:35:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4666000000000002e-05 2023-10-20T03:35:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370634+0000", 2023-10-20T03:35:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1866999999999998e-05 2023-10-20T03:35:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.247 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370657+0000", 2023-10-20T03:35:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3010000000000002e-05 2023-10-20T03:35:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.248 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.375481+0000", 2023-10-20T03:35:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0048233099999999999 2023-10-20T03:35:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.249 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.375508+0000", 2023-10-20T03:35:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.703e-05 2023-10-20T03:35:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.381064+0000", 2023-10-20T03:35:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055565750000000002 2023-10-20T03:35:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.381117+0000", 2023-10-20T03:35:46.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2833000000000001e-05 2023-10-20T03:35:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.252 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.0s6 44/31 ECSubWrite(tid=18, reqid=client.4570.0:323, at_version=44'9, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:35:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.361041+0000", 2023-10-20T03:35:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 19.853111187, 2023-10-20T03:35:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015954300000000001, 2023-10-20T03:35:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.253 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361041+0000", 2023-10-20T03:35:46.254 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361041+0000", 2023-10-20T03:35:46.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361041+0000", 2023-10-20T03:35:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2699999999999998e-07 2023-10-20T03:35:46.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361502+0000", 2023-10-20T03:35:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046024999999999999 2023-10-20T03:35:46.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361532+0000", 2023-10-20T03:35:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0286000000000001e-05 2023-10-20T03:35:46.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361548+0000", 2023-10-20T03:35:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5797000000000001e-05 2023-10-20T03:35:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361674+0000", 2023-10-20T03:35:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012640500000000001 2023-10-20T03:35:46.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361693+0000", 2023-10-20T03:35:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9066999999999999e-05 2023-10-20T03:35:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.376883+0000", 2023-10-20T03:35:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015189569 2023-10-20T03:35:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.262 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.376995+0000", 2023-10-20T03:35:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112099 2023-10-20T03:35:46.263 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.0s6 44/31 ECSubWrite(tid=19, reqid=client.4570.0:324, at_version=44'10, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:35:46.264 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.387754+0000", 2023-10-20T03:35:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 19.826397271000001, 2023-10-20T03:35:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020931727000000001, 2023-10-20T03:35:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.265 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387754+0000", 2023-10-20T03:35:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.266 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387754+0000", 2023-10-20T03:35:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387756+0000", 2023-10-20T03:35:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.573e-06 2023-10-20T03:35:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.268 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388025+0000", 2023-10-20T03:35:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026924199999999999 2023-10-20T03:35:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388058+0000", 2023-10-20T03:35:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2339999999999999e-05 2023-10-20T03:35:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.270 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388073+0000", 2023-10-20T03:35:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5489e-05 2023-10-20T03:35:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.271 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388242+0000", 2023-10-20T03:35:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016881200000000001 2023-10-20T03:35:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.272 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388268+0000", 2023-10-20T03:35:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6169999999999998e-05 2023-10-20T03:35:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.273 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.408608+0000", 2023-10-20T03:35:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020340272999999999 2023-10-20T03:35:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.274 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.408686+0000", 2023-10-20T03:35:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7828000000000001e-05 2023-10-20T03:35:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.275 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.0s6 44/31 ECSubWrite(tid=20, reqid=client.4570.0:325, at_version=44'11, trim_to=0'0, roll_forward_to=44'9))", 2023-10-20T03:35:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.398275+0000", 2023-10-20T03:35:46.276 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 19.815876273000001, 2023-10-20T03:35:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012514033000000001, 2023-10-20T03:35:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.277 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398275+0000", 2023-10-20T03:35:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.278 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398275+0000", 2023-10-20T03:35:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.279 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398278+0000", 2023-10-20T03:35:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.046e-06 2023-10-20T03:35:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.280 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398375+0000", 2023-10-20T03:35:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7816e-05 2023-10-20T03:35:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.281 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398415+0000", 2023-10-20T03:35:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9660000000000003e-05 2023-10-20T03:35:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.282 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398438+0000", 2023-10-20T03:35:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2756999999999999e-05 2023-10-20T03:35:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.283 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398613+0000", 2023-10-20T03:35:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000175415 2023-10-20T03:35:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398638+0000", 2023-10-20T03:35:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4805999999999999e-05 2023-10-20T03:35:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.410452+0000", 2023-10-20T03:35:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.011813784000000001 2023-10-20T03:35:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.286 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.410790+0000", 2023-10-20T03:35:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00033774899999999998 2023-10-20T03:35:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.287 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s4 44/31 ECSubWrite(tid=22, reqid=client.4570.0:340, at_version=44'12, trim_to=0'0, roll_forward_to=43'10))", 2023-10-20T03:35:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.399701+0000", 2023-10-20T03:35:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 19.814451034000001, 2023-10-20T03:35:46.288 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0061945460000000004, 2023-10-20T03:35:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.289 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399701+0000", 2023-10-20T03:35:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.290 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399701+0000", 2023-10-20T03:35:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.291 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399701+0000", 2023-10-20T03:35:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.06e-07 2023-10-20T03:35:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399806+0000", 2023-10-20T03:35:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010474700000000001 2023-10-20T03:35:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399836+0000", 2023-10-20T03:35:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0059000000000001e-05 2023-10-20T03:35:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399858+0000", 2023-10-20T03:35:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1829999999999999e-05 2023-10-20T03:35:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.295 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.400095+0000", 2023-10-20T03:35:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00023654500000000001 2023-10-20T03:35:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.400120+0000", 2023-10-20T03:35:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5829999999999998e-05 2023-10-20T03:35:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.405795+0000", 2023-10-20T03:35:46.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0056742069999999997 2023-10-20T03:35:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.405895+0000", 2023-10-20T03:35:46.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100722 2023-10-20T03:35:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.0s6 44/31 ECSubWrite(tid=21, reqid=client.4570.0:326, at_version=44'12, trim_to=0'0, roll_forward_to=44'9))", 2023-10-20T03:35:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.403096+0000", 2023-10-20T03:35:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 19.811056232999999, 2023-10-20T03:35:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0086307510000000007, 2023-10-20T03:35:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.300 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403096+0000", 2023-10-20T03:35:46.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403096+0000", 2023-10-20T03:35:46.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403098+0000", 2023-10-20T03:35:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9870000000000002e-06 2023-10-20T03:35:46.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403150+0000", 2023-10-20T03:35:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2271999999999997e-05 2023-10-20T03:35:46.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403185+0000", 2023-10-20T03:35:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5373999999999997e-05 2023-10-20T03:35:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403208+0000", 2023-10-20T03:35:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.334e-05 2023-10-20T03:35:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.407733+0000", 2023-10-20T03:35:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.004524312 2023-10-20T03:35:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.407756+0000", 2023-10-20T03:35:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2991000000000001e-05 2023-10-20T03:35:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.411611+0000", 2023-10-20T03:35:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0038554990000000001 2023-10-20T03:35:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.411726+0000", 2023-10-20T03:35:46.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011497600000000001 2023-10-20T03:35:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.0s6 49/31 ECSubWrite(tid=39, reqid=client.4570.0:390, at_version=49'23, trim_to=0'0, roll_forward_to=47'21))", 2023-10-20T03:35:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.413616+0000", 2023-10-20T03:35:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 14.800535976999999, 2023-10-20T03:35:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.02288652, 2023-10-20T03:35:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.311 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.413616+0000", 2023-10-20T03:35:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.413616+0000", 2023-10-20T03:35:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.413617+0000", 2023-10-20T03:35:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2600000000000002e-07 2023-10-20T03:35:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.414748+0000", 2023-10-20T03:35:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0011315520000000001 2023-10-20T03:35:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.414773+0000", 2023-10-20T03:35:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.478e-05 2023-10-20T03:35:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.414788+0000", 2023-10-20T03:35:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4926000000000001e-05 2023-10-20T03:35:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.415003+0000", 2023-10-20T03:35:46.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021510499999999999 2023-10-20T03:35:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.415029+0000", 2023-10-20T03:35:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6429999999999999e-05 2023-10-20T03:35:46.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.436443+0000", 2023-10-20T03:35:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021413835999999999 2023-10-20T03:35:46.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.436502+0000", 2023-10-20T03:35:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9165e-05 2023-10-20T03:35:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.320 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s4 53/31 ECSubWrite(tid=33, reqid=osd.8.0:33, at_version=53'24, trim_to=0'0, roll_forward_to=52'22))", 2023-10-20T03:35:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:35.432252+0000", 2023-10-20T03:35:46.321 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.781900025000001, 2023-10-20T03:35:46.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0072446250000000002, 2023-10-20T03:35:46.322 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.322 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.322 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.432252+0000", 2023-10-20T03:35:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.432252+0000", 2023-10-20T03:35:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.432252+0000", 2023-10-20T03:35:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6700000000000003e-07 2023-10-20T03:35:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.325 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.432312+0000", 2023-10-20T03:35:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9661999999999999e-05 2023-10-20T03:35:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.326 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.432343+0000", 2023-10-20T03:35:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1324e-05 2023-10-20T03:35:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.432368+0000", 2023-10-20T03:35:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4366e-05 2023-10-20T03:35:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.432917+0000", 2023-10-20T03:35:46.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00054973299999999995 2023-10-20T03:35:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.432947+0000", 2023-10-20T03:35:46.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9266000000000001e-05 2023-10-20T03:35:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.439361+0000", 2023-10-20T03:35:46.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0064147079999999999 2023-10-20T03:35:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.439496+0000", 2023-10-20T03:35:46.331 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013499900000000001 2023-10-20T03:35:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.3.22:2 3.as0 3.82f684ca (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:35:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:35.440266+0000", 2023-10-20T03:35:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.773885482000001, 2023-10-20T03:35:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.98271485700000005, 2023-10-20T03:35:46.333 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440266+0000", 2023-10-20T03:35:46.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440266+0000", 2023-10-20T03:35:46.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440269+0000", 2023-10-20T03:35:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4930000000000001e-06 2023-10-20T03:35:46.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440282+0000", 2023-10-20T03:35:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.33e-05 2023-10-20T03:35:46.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440310+0000", 2023-10-20T03:35:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7569999999999999e-05 2023-10-20T03:35:46.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440332+0000", 2023-10-20T03:35:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2633000000000002e-05 2023-10-20T03:35:46.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440473+0000", 2023-10-20T03:35:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014078499999999999 2023-10-20T03:35:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2023-10-20T03:35:46.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440478+0000", 2023-10-20T03:35:46.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3530000000000002e-06 2023-10-20T03:35:46.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.422959+0000", 2023-10-20T03:35:46.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.98248045299999998 2023-10-20T03:35:46.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.343 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.422981+0000", 2023-10-20T03:35:46.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2269999999999999e-05 2023-10-20T03:35:46.343 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.343 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.343 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.344 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.0s6 54/31 ECSubWrite(tid=65, reqid=client.4570.0:418, at_version=54'32, trim_to=0'0, roll_forward_to=51'31))", 2023-10-20T03:35:46.344 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:36.458901+0000", 2023-10-20T03:35:46.344 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 9.7552509809999997, 2023-10-20T03:35:46.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084431630000000001, 2023-10-20T03:35:46.345 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.345 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.345 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.458901+0000", 2023-10-20T03:35:46.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.458901+0000", 2023-10-20T03:35:46.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.458901+0000", 2023-10-20T03:35:46.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2599999999999999e-07 2023-10-20T03:35:46.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.348 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.348 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.459275+0000", 2023-10-20T03:35:46.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00037360099999999999 2023-10-20T03:35:46.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.349 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.349 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.459288+0000", 2023-10-20T03:35:46.349 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3621999999999999e-05 2023-10-20T03:35:46.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.459301+0000", 2023-10-20T03:35:46.350 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3257e-05 2023-10-20T03:35:46.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.351 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.351 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.459413+0000", 2023-10-20T03:35:46.351 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111369 2023-10-20T03:35:46.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.351 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.352 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.459424+0000", 2023-10-20T03:35:46.352 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0983e-05 2023-10-20T03:35:46.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.352 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.352 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.467308+0000", 2023-10-20T03:35:46.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0078843260000000005 2023-10-20T03:35:46.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.353 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.467344+0000", 2023-10-20T03:35:46.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5879e-05 2023-10-20T03:35:46.354 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.354 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.354 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.355 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.0s6 54/31 ECSubWrite(tid=67, reqid=client.4570.0:420, at_version=54'34, trim_to=0'0, roll_forward_to=54'32))", 2023-10-20T03:35:46.355 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:36.487627+0000", 2023-10-20T03:35:46.355 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 9.726524607, 2023-10-20T03:35:46.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0061264270000000003, 2023-10-20T03:35:46.356 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.356 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.356 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.356 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.356 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.487627+0000", 2023-10-20T03:35:46.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.487627+0000", 2023-10-20T03:35:46.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.487628+0000", 2023-10-20T03:35:46.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3099999999999996e-07 2023-10-20T03:35:46.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.487700+0000", 2023-10-20T03:35:46.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2356999999999999e-05 2023-10-20T03:35:46.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.487718+0000", 2023-10-20T03:35:46.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7099999999999999e-05 2023-10-20T03:35:46.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.487728+0000", 2023-10-20T03:35:46.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0169999999999999e-05 2023-10-20T03:35:46.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.362 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.362 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.487812+0000", 2023-10-20T03:35:46.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4295000000000005e-05 2023-10-20T03:35:46.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.487820+0000", 2023-10-20T03:35:46.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8480000000000001e-06 2023-10-20T03:35:46.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.493668+0000", 2023-10-20T03:35:46.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0058484929999999997 2023-10-20T03:35:46.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.493754+0000", 2023-10-20T03:35:46.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5233000000000002e-05 2023-10-20T03:35:46.365 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.365 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.365 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.366 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s4 60/31 ECSubWrite(tid=39, reqid=osd.8.0:39, at_version=60'30, trim_to=0'0, roll_forward_to=56'28))", 2023-10-20T03:35:46.366 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.841850+0000", 2023-10-20T03:35:46.366 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.3723013139999996, 2023-10-20T03:35:46.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0063416189999999997, 2023-10-20T03:35:46.367 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.367 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.367 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.841850+0000", 2023-10-20T03:35:46.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.841850+0000", 2023-10-20T03:35:46.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.841851+0000", 2023-10-20T03:35:46.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3699999999999999e-07 2023-10-20T03:35:46.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.370 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.841911+0000", 2023-10-20T03:35:46.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9243000000000002e-05 2023-10-20T03:35:46.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.841959+0000", 2023-10-20T03:35:46.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8696000000000003e-05 2023-10-20T03:35:46.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.842009+0000", 2023-10-20T03:35:46.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9336000000000003e-05 2023-10-20T03:35:46.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.842248+0000", 2023-10-20T03:35:46.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00023933900000000001 2023-10-20T03:35:46.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.842280+0000", 2023-10-20T03:35:46.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2106000000000001e-05 2023-10-20T03:35:46.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.848076+0000", 2023-10-20T03:35:46.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057958829999999999 2023-10-20T03:35:46.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.848192+0000", 2023-10-20T03:35:46.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000116179 2023-10-20T03:35:46.376 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.377 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.377 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.377 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.9s5 60/31 ECSubWrite(tid=45, reqid=osd.7.0:45, at_version=60'32, trim_to=0'0, roll_forward_to=56'30))", 2023-10-20T03:35:46.378 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.858116+0000", 2023-10-20T03:35:46.378 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.3560353999999997, 2023-10-20T03:35:46.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0064152940000000002, 2023-10-20T03:35:46.378 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.378 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.379 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.858116+0000", 2023-10-20T03:35:46.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.858116+0000", 2023-10-20T03:35:46.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.858118+0000", 2023-10-20T03:35:46.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.623e-06 2023-10-20T03:35:46.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.858203+0000", 2023-10-20T03:35:46.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5018999999999994e-05 2023-10-20T03:35:46.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.858236+0000", 2023-10-20T03:35:46.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2753999999999998e-05 2023-10-20T03:35:46.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.858261+0000", 2023-10-20T03:35:46.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5185999999999999e-05 2023-10-20T03:35:46.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.858412+0000", 2023-10-20T03:35:46.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000151502 2023-10-20T03:35:46.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.858430+0000", 2023-10-20T03:35:46.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8012999999999998e-05 2023-10-20T03:35:46.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.864446+0000", 2023-10-20T03:35:46.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0060159549999999999 2023-10-20T03:35:46.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.864532+0000", 2023-10-20T03:35:46.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5241999999999995e-05 2023-10-20T03:35:46.388 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.388 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.388 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.389 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:466 3.as0 3:55779659:::smithi17039300-22:head [create,setxattr \\\\jePWq]ZOWNpZOUZYVPc^lpj`fUUtsQVY\\NhUQ] (1719) in=40b,setxattr aoaWl[dPlh\\XpV^jtfVhPOYqn\\NhXU[najjbRNWd (10033) in=40b,setxattr `V_nomOPOmgVgNrNl`jNVs_kSmRdSl^ioTPc\\^eY (19716) in=40b,setxattr lecebNUs]cdSasSblV`sWfcidcPc^bU (279) in=31b,setxattr cS]cSWa_lh^jVbWphdgpXmgYYcnank`UpmobcZUa (4768) in=40b,setxattr hmNTeiNaYhNQOPaRafS\\N[Q\\lni (279) in=27b,setxattr sVZOOgbVN`ejVfhSRZcaTtqnfdgPbpaYeNQbSf`c (9120) in=40b,setxattr dobiNs^RXTPjhaQtOaipXXfdWQYafcajclaW`TiO (1050) in=40b,setxattr nqoachfoWSY`OSgs_VoZTbhe]egYaQiZl[RTphVr (1821) in=40b,setxattr g\\scUhe]\\_jRp^SnXga_\\VtssTTbbaj^cSq\\ffmW (595) in=40b,setxattr UXV`NXQYomt]eTei[ksqfOr_cm (279) in=26b,setxattr Y_]abT\\^UYakbfWl]\\dTibZaSYj^aa^oUQtWpaQa (771) in=40b,setxattr ]Xga]WS^XrRqqrQSfTaoOgSYam_`mPmNXg`tNYmT (1789) in=40b,setxattr WZUeZrcUPqodUr`\\UmkgYpXpr[W^ceZRfhO\\nRaj (17824) in=40b,setxattr \\pXoTRPmQWkcio`^a^X`gNTan_Rcop[gm_ecWliQ (946) in=40b,setxattr NnZhqU`cnT_edhVicNgm`a[XR\\ZOoNaZshtl[Paq (15156) in=40b,setxattr lhackQeZc^aYSZgjq][O[rfV\\cgsZ[tah_Q_cX\\h (8420) in=40b,setxattr PWN]cjYVneeRNnlYgcijosg]e^_oTP_[YlT[_gX^ (1649) in=40b,setxattr WtRNQ`NYWWmoWpjVqhb]\\apOq]RTUbOfshmUmtTR (8628) in=40b,setxattr hmX_UbmcY[ROefRbcXXg\\okbS`bTONmsnTN[ceSt (19508) in=40b] snapc 19=[19] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2023-10-20T03:35:46.389 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.906742+0000", 2023-10-20T03:35:46.389 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.307409958, 2023-10-20T03:35:46.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013943011999999999, 2023-10-20T03:35:46.390 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.390 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:46.390 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:46.390 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:46.391 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:46.391 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 466 2023-10-20T03:35:46.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.391 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.906742+0000", 2023-10-20T03:35:46.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.906742+0000", 2023-10-20T03:35:46.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.906744+0000", 2023-10-20T03:35:46.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0339999999999999e-06 2023-10-20T03:35:46.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907135+0000", 2023-10-20T03:35:46.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000390751 2023-10-20T03:35:46.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907157+0000", 2023-10-20T03:35:46.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2609e-05 2023-10-20T03:35:46.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907172+0000", 2023-10-20T03:35:46.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4698000000000001e-05 2023-10-20T03:35:46.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907285+0000", 2023-10-20T03:35:46.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000113263 2023-10-20T03:35:46.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:46.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907535+0000", 2023-10-20T03:35:46.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000249683 2023-10-20T03:35:46.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.909660+0000", 2023-10-20T03:35:46.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021251030000000001 2023-10-20T03:35:46.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.915771+0000", 2023-10-20T03:35:46.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0061108619999999999 2023-10-20T03:35:46.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:46.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.920228+0000", 2023-10-20T03:35:46.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0044575500000000002 2023-10-20T03:35:46.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.920685+0000", 2023-10-20T03:35:46.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00045645899999999999 2023-10-20T03:35:46.402 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.403 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.403 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.403 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:467 3.as0 3:51752c29:::smithi17039300-1:head [read 0~1327104,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-10-20T03:35:46.404 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.907523+0000", 2023-10-20T03:35:46.404 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.3066284509999999, 2023-10-20T03:35:46.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0122909, 2023-10-20T03:35:46.404 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.404 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:35:46.405 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:46.405 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:46.405 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:46.405 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 467 2023-10-20T03:35:46.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.405 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907523+0000", 2023-10-20T03:35:46.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907523+0000", 2023-10-20T03:35:46.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907525+0000", 2023-10-20T03:35:46.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5149999999999999e-06 2023-10-20T03:35:46.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907536+0000", 2023-10-20T03:35:46.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.116e-05 2023-10-20T03:35:46.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907554+0000", 2023-10-20T03:35:46.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8094e-05 2023-10-20T03:35:46.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907566+0000", 2023-10-20T03:35:46.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2249000000000001e-05 2023-10-20T03:35:46.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.911271+0000", 2023-10-20T03:35:46.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003704897 2023-10-20T03:35:46.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:46.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.911463+0000", 2023-10-20T03:35:46.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019189700000000001 2023-10-20T03:35:46.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.919814+0000", 2023-10-20T03:35:46.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0083510879999999996 2023-10-20T03:35:46.414 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.414 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.414 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.415 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:468 3.as0 3:51752c29:::smithi17039300-1:head [read 0~1327104,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-10-20T03:35:46.415 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.907631+0000", 2023-10-20T03:35:46.415 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.3065203859999999, 2023-10-20T03:35:46.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016440132, 2023-10-20T03:35:46.416 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.416 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:35:46.416 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:46.416 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:46.416 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:46.416 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 468 2023-10-20T03:35:46.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.417 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907631+0000", 2023-10-20T03:35:46.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907631+0000", 2023-10-20T03:35:46.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907632+0000", 2023-10-20T03:35:46.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.048e-06 2023-10-20T03:35:46.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907644+0000", 2023-10-20T03:35:46.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1555e-05 2023-10-20T03:35:46.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907677+0000", 2023-10-20T03:35:46.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3207999999999998e-05 2023-10-20T03:35:46.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907691+0000", 2023-10-20T03:35:46.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3698e-05 2023-10-20T03:35:46.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.911974+0000", 2023-10-20T03:35:46.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0042835690000000001 2023-10-20T03:35:46.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:46.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.912070+0000", 2023-10-20T03:35:46.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5915999999999994e-05 2023-10-20T03:35:46.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.924072+0000", 2023-10-20T03:35:46.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012001138 2023-10-20T03:35:46.425 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.425 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.425 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.426 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:469 3.as0 3:51752c29:::smithi17039300-1:head [read 0~1327104,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-10-20T03:35:46.426 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.907719+0000", 2023-10-20T03:35:46.426 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.3064330450000003, 2023-10-20T03:35:46.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018910997999999998, 2023-10-20T03:35:46.426 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.427 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:35:46.427 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:46.427 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:46.427 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:46.427 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 469 2023-10-20T03:35:46.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.428 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907719+0000", 2023-10-20T03:35:46.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907719+0000", 2023-10-20T03:35:46.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907720+0000", 2023-10-20T03:35:46.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.006e-06 2023-10-20T03:35:46.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907729+0000", 2023-10-20T03:35:46.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0520000000000007e-06 2023-10-20T03:35:46.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907743+0000", 2023-10-20T03:35:46.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4258e-05 2023-10-20T03:35:46.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907757+0000", 2023-10-20T03:35:46.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3755e-05 2023-10-20T03:35:46.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.912479+0000", 2023-10-20T03:35:46.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0047223079999999997 2023-10-20T03:35:46.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:46.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.912642+0000", 2023-10-20T03:35:46.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000162795 2023-10-20T03:35:46.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.926630+0000", 2023-10-20T03:35:46.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013987823999999999 2023-10-20T03:35:46.436 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.436 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.436 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.436 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpReadReply(3.as0 60/56 ECSubReadReply(tid=4, attrs_read=0))", 2023-10-20T03:35:46.437 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.918429+0000", 2023-10-20T03:35:46.437 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2957227250000001, 2023-10-20T03:35:46.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0082862909999999994, 2023-10-20T03:35:46.437 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.437 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:46.438 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.918429+0000", 2023-10-20T03:35:46.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.918429+0000", 2023-10-20T03:35:46.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.918429+0000", 2023-10-20T03:35:46.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8700000000000001e-07 2023-10-20T03:35:46.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.918713+0000", 2023-10-20T03:35:46.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028325600000000001 2023-10-20T03:35:46.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.918723+0000", 2023-10-20T03:35:46.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0043e-05 2023-10-20T03:35:46.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.918728+0000", 2023-10-20T03:35:46.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0100000000000003e-06 2023-10-20T03:35:46.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.924216+0000", 2023-10-20T03:35:46.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0054880379999999998 2023-10-20T03:35:46.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.926715+0000", 2023-10-20T03:35:46.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0024995569999999999 2023-10-20T03:35:46.445 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.445 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.446 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.446 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.446 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.446 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:46.448 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-10-20T03:35:46.448 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:46.444+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:46.548 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:46.545+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:46.626 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:46.627 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:46.627 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:35:46.627 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:46.641 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-10-20T03:35:46.650 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:46.649+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:46.751 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:46.748+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:46.760 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:46.760 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:35:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:35:46.761 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:35:46.761 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:46.776 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-10-20T03:35:46.851 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:46.849+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:46.874 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.3 primary-affinity to 0.958457 (8628132) 2023-10-20T03:35:46.882 INFO:tasks.rados.rados.0.smithi170.stdout:224: done (2 left) 2023-10-20T03:35:46.882 INFO:tasks.rados.rados.0.smithi170.stdout:225: read oid 32 snap -1 2023-10-20T03:35:46.883 INFO:tasks.rados.rados.0.smithi170.stdout:225: expect (ObjNum 104 snap 13 seq_num 104) 2023-10-20T03:35:46.883 INFO:tasks.rados.rados.0.smithi170.stdout:226: snap_remove snap 14 2023-10-20T03:35:46.894 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-10-20T03:35:46.915 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:35:46.915 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:35:46.916 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:35:46.916 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2023-10-20T03:35:46.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.916 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:306 3.1s0 3:8d0c2319:::smithi17039300-31:head [append 0~557056] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:46.916 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.325425+0000", 2023-10-20T03:35:46.917 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.588231151999999, 2023-10-20T03:35:46.917 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040045958999999999, 2023-10-20T03:35:46.917 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.917 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:46.917 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:46.917 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:46.918 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:46.918 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 306 2023-10-20T03:35:46.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.918 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.919 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.919 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.325425+0000", 2023-10-20T03:35:46.919 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.919 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.920 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.325425+0000", 2023-10-20T03:35:46.921 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.922 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.922 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.325427+0000", 2023-10-20T03:35:46.922 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.723e-06 2023-10-20T03:35:46.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.922 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.923 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.326339+0000", 2023-10-20T03:35:46.923 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00091144999999999996 2023-10-20T03:35:46.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.923 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.924 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.326373+0000", 2023-10-20T03:35:46.924 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4613000000000002e-05 2023-10-20T03:35:46.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.924 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.924 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.326391+0000", 2023-10-20T03:35:46.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8029e-05 2023-10-20T03:35:46.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.925 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.925 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.326524+0000", 2023-10-20T03:35:46.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013269900000000001 2023-10-20T03:35:46.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.926 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:46.926 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.326702+0000", 2023-10-20T03:35:46.926 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000178083 2023-10-20T03:35:46.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.927 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.927 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.927 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.328888+0000", 2023-10-20T03:35:46.927 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021861900000000002 2023-10-20T03:35:46.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.927 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.928 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.928 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.357500+0000", 2023-10-20T03:35:46.928 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.028611365 2023-10-20T03:35:46.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.928 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:46.929 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365350+0000", 2023-10-20T03:35:46.929 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0078497710000000002 2023-10-20T03:35:46.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.929 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.929 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365471+0000", 2023-10-20T03:35:46.930 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000121036 2023-10-20T03:35:46.930 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.930 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.930 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.931 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:308 3.1s0 3:8d0c2319:::smithi17039300-31:head [append 0~98304] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:46.931 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.338932+0000", 2023-10-20T03:35:46.931 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.574724144000001, 2023-10-20T03:35:46.931 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033689486999999997, 2023-10-20T03:35:46.931 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.932 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:46.932 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:46.932 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:46.932 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:46.932 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 308 2023-10-20T03:35:46.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.933 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.933 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.338932+0000", 2023-10-20T03:35:46.933 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.934 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.934 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.338932+0000", 2023-10-20T03:35:46.934 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.935 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.935 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.338934+0000", 2023-10-20T03:35:46.935 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0770000000000002e-06 2023-10-20T03:35:46.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.936 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339139+0000", 2023-10-20T03:35:46.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020497500000000001 2023-10-20T03:35:46.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339174+0000", 2023-10-20T03:35:46.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5583000000000001e-05 2023-10-20T03:35:46.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.937 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339194+0000", 2023-10-20T03:35:46.938 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9245999999999998e-05 2023-10-20T03:35:46.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.938 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.938 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.353975+0000", 2023-10-20T03:35:46.939 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014781949000000001 2023-10-20T03:35:46.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.939 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:46.939 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.354190+0000", 2023-10-20T03:35:46.940 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021416600000000001 2023-10-20T03:35:46.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.940 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.940 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.355201+0000", 2023-10-20T03:35:46.940 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0010116719999999999 2023-10-20T03:35:46.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.359820+0000", 2023-10-20T03:35:46.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0046182380000000002 2023-10-20T03:35:46.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.942 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:46.942 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.372433+0000", 2023-10-20T03:35:46.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012613509 2023-10-20T03:35:46.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.943 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.943 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.372621+0000", 2023-10-20T03:35:46.943 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018807200000000001 2023-10-20T03:35:46.943 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.943 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.944 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.944 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:309 3.1s0 3:8d0c2319:::smithi17039300-31:head [setxattr _header (56) in=7b] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:46.944 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.339226+0000", 2023-10-20T03:35:46.945 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.574430259, 2023-10-20T03:35:46.945 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035095548999999997, 2023-10-20T03:35:46.945 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.945 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:46.945 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:46.945 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:46.946 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:46.946 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 309 2023-10-20T03:35:46.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.946 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.946 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.947 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339226+0000", 2023-10-20T03:35:46.947 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.947 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.947 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339226+0000", 2023-10-20T03:35:46.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.948 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.948 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339227+0000", 2023-10-20T03:35:46.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.376e-06 2023-10-20T03:35:46.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.949 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.949 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.949 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339236+0000", 2023-10-20T03:35:46.949 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2799999999999992e-06 2023-10-20T03:35:46.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.950 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.950 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339256+0000", 2023-10-20T03:35:46.950 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9524000000000001e-05 2023-10-20T03:35:46.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.951 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.951 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.951 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339270+0000", 2023-10-20T03:35:46.951 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.47e-05 2023-10-20T03:35:46.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.951 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.952 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.952 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.357964+0000", 2023-10-20T03:35:46.952 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018693279 2023-10-20T03:35:46.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.952 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.952 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:46.953 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.358151+0000", 2023-10-20T03:35:46.953 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018687200000000001 2023-10-20T03:35:46.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.953 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.953 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.954 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.359027+0000", 2023-10-20T03:35:46.954 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087691600000000005 2023-10-20T03:35:46.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.954 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.954 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.955 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.361861+0000", 2023-10-20T03:35:46.955 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0028334319999999999 2023-10-20T03:35:46.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.955 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.955 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:46.955 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.374062+0000", 2023-10-20T03:35:46.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012201372 2023-10-20T03:35:46.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.956 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.956 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.374321+0000", 2023-10-20T03:35:46.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025879800000000002 2023-10-20T03:35:46.957 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.957 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.957 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.958 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:310 3.1s0 3:8d0c2319:::smithi17039300-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:46.958 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.339292+0000", 2023-10-20T03:35:46.958 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.574363680000001, 2023-10-20T03:35:46.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.042306519000000001, 2023-10-20T03:35:46.958 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.958 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:35:46.959 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:46.959 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:46.959 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:46.959 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 310 2023-10-20T03:35:46.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.959 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.960 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.960 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.960 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339292+0000", 2023-10-20T03:35:46.960 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.960 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.961 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339292+0000", 2023-10-20T03:35:46.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339294+0000", 2023-10-20T03:35:46.962 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4130000000000001e-06 2023-10-20T03:35:46.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.962 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339300+0000", 2023-10-20T03:35:46.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0499999999999997e-06 2023-10-20T03:35:46.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.963 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.963 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.963 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339334+0000", 2023-10-20T03:35:46.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4513e-05 2023-10-20T03:35:46.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.964 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.964 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.964 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.339372+0000", 2023-10-20T03:35:46.964 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.769e-05 2023-10-20T03:35:46.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.965 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.965 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.359539+0000", 2023-10-20T03:35:46.965 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020167241999999998 2023-10-20T03:35:46.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:46.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.359694+0000", 2023-10-20T03:35:46.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000154585 2023-10-20T03:35:46.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.967 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.376927+0000", 2023-10-20T03:35:46.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017233637 2023-10-20T03:35:46.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.967 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:46.968 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.377208+0000", 2023-10-20T03:35:46.968 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028108099999999997 2023-10-20T03:35:46.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.968 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.968 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.969 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.381599+0000", 2023-10-20T03:35:46.969 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043903079999999999 2023-10-20T03:35:46.969 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.969 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.969 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.969 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.970 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:312 3.6s0 3:62cf25fd:::smithi17039300-14:head [append 0~425984] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:46.970 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.360818+0000", 2023-10-20T03:35:46.970 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.552837739000001, 2023-10-20T03:35:46.970 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.035316588000000003, 2023-10-20T03:35:46.971 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.971 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:46.971 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:46.971 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:46.971 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:46.971 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 312 2023-10-20T03:35:46.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.972 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.972 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.972 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.972 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.360818+0000", 2023-10-20T03:35:46.973 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.973 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.973 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.973 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.360818+0000", 2023-10-20T03:35:46.973 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.974 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.974 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.974 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.360820+0000", 2023-10-20T03:35:46.974 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1969999999999999e-06 2023-10-20T03:35:46.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.975 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.975 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.975 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.361437+0000", 2023-10-20T03:35:46.975 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00061634199999999997 2023-10-20T03:35:46.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.976 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.976 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.361474+0000", 2023-10-20T03:35:46.976 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7735999999999999e-05 2023-10-20T03:35:46.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.977 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.361497+0000", 2023-10-20T03:35:46.977 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2246999999999999e-05 2023-10-20T03:35:46.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.977 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369097+0000", 2023-10-20T03:35:46.978 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0076008430000000004 2023-10-20T03:35:46.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.978 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.978 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:46.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369369+0000", 2023-10-20T03:35:46.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000271721 2023-10-20T03:35:46.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.371397+0000", 2023-10-20T03:35:46.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0020282519999999999 2023-10-20T03:35:46.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.980 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.980 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.980 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.395428+0000", 2023-10-20T03:35:46.980 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024030335 2023-10-20T03:35:46.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.981 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.981 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:46.981 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.395694+0000", 2023-10-20T03:35:46.981 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026599799999999998 2023-10-20T03:35:46.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.981 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.982 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.982 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.396135+0000", 2023-10-20T03:35:46.982 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044091699999999998 2023-10-20T03:35:46.982 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.982 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.982 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.983 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:314 3.6s0 3:62cf25fd:::smithi17039300-14:head [setxattr _header (56) in=7b] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:46.983 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.364991+0000", 2023-10-20T03:35:46.983 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.548664646999999, 2023-10-20T03:35:46.984 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032796803999999999, 2023-10-20T03:35:46.984 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.984 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:46.984 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:46.984 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:46.985 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:46.985 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 314 2023-10-20T03:35:46.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.985 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.985 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.986 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.364991+0000", 2023-10-20T03:35:46.986 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.986 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.986 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.364991+0000", 2023-10-20T03:35:46.987 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.987 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:46.987 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.364993+0000", 2023-10-20T03:35:46.987 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5239999999999999e-06 2023-10-20T03:35:46.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.988 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.988 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:46.988 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365005+0000", 2023-10-20T03:35:46.988 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2033e-05 2023-10-20T03:35:46.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.989 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:46.989 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365030+0000", 2023-10-20T03:35:46.989 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5507000000000002e-05 2023-10-20T03:35:46.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.990 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:46.990 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365047+0000", 2023-10-20T03:35:46.990 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6753e-05 2023-10-20T03:35:46.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.990 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.990 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:46.991 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.393279+0000", 2023-10-20T03:35:46.991 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.028232090000000001 2023-10-20T03:35:46.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.991 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.991 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:46.991 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.393435+0000", 2023-10-20T03:35:46.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015599900000000001 2023-10-20T03:35:46.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.992 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.992 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:46.992 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.394369+0000", 2023-10-20T03:35:46.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00093351700000000003 2023-10-20T03:35:46.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.993 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.993 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:46.993 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.396532+0000", 2023-10-20T03:35:46.993 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021638640000000002 2023-10-20T03:35:46.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.993 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.994 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:46.994 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.397496+0000", 2023-10-20T03:35:46.994 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00096351599999999998 2023-10-20T03:35:46.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.994 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.995 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:46.995 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.397788+0000", 2023-10-20T03:35:46.995 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00029200100000000001 2023-10-20T03:35:46.995 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.995 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:46.995 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:46.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.996 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.996 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:315 3.6s0 3:62cf25fd:::smithi17039300-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:46.996 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.365076+0000", 2023-10-20T03:35:46.996 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.548580250000001, 2023-10-20T03:35:46.996 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.036384568999999999, 2023-10-20T03:35:46.997 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:46.997 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:35:46.997 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:46.997 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:46.997 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:46.997 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 315 2023-10-20T03:35:46.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.998 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:46.998 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.998 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:46.998 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365076+0000", 2023-10-20T03:35:46.998 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.999 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:46.999 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:46.999 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365076+0000", 2023-10-20T03:35:46.999 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:46.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:46.999 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.000 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365077+0000", 2023-10-20T03:35:47.000 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.426e-06 2023-10-20T03:35:47.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.001 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365084+0000", 2023-10-20T03:35:47.001 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2099999999999996e-06 2023-10-20T03:35:47.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.001 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.001 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365105+0000", 2023-10-20T03:35:47.002 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0633e-05 2023-10-20T03:35:47.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.002 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.002 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.365119+0000", 2023-10-20T03:35:47.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.412e-05 2023-10-20T03:35:47.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.003 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.003 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.396242+0000", 2023-10-20T03:35:47.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031122606000000001 2023-10-20T03:35:47.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:47.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.396417+0000", 2023-10-20T03:35:47.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017512199999999999 2023-10-20T03:35:47.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.005 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.398627+0000", 2023-10-20T03:35:47.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022101170000000002 2023-10-20T03:35:47.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.006 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:47.006 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.398910+0000", 2023-10-20T03:35:47.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028353800000000001 2023-10-20T03:35:47.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.006 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.006 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.007 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.401460+0000", 2023-10-20T03:35:47.007 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025497969999999999 2023-10-20T03:35:47.007 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.007 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.007 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.008 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds5 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:35:47.008 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:30.393790+0000", 2023-10-20T03:35:47.008 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 16.519865391, 2023-10-20T03:35:47.008 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.037929888000000002, 2023-10-20T03:35:47.008 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:47.009 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:35:47.009 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:47.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.009 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:47.009 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.393790+0000", 2023-10-20T03:35:47.010 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.010 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:47.010 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.393790+0000", 2023-10-20T03:35:47.010 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.011 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.011 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.393792+0000", 2023-10-20T03:35:47.011 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.53e-06 2023-10-20T03:35:47.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394998+0000", 2023-10-20T03:35:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001205734 2023-10-20T03:35:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.013 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.013 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.395041+0000", 2023-10-20T03:35:47.013 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3037000000000001e-05 2023-10-20T03:35:47.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.013 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.014 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.395065+0000", 2023-10-20T03:35:47.014 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4056000000000001e-05 2023-10-20T03:35:47.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.014 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.014 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.395270+0000", 2023-10-20T03:35:47.015 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020527499999999999 2023-10-20T03:35:47.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.015 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:47.015 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.395298+0000", 2023-10-20T03:35:47.015 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8167e-05 2023-10-20T03:35:47.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.016 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:47.016 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.431637+0000", 2023-10-20T03:35:47.016 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.036339123000000001 2023-10-20T03:35:47.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.017 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.017 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.431720+0000", 2023-10-20T03:35:47.017 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2966000000000005e-05 2023-10-20T03:35:47.017 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.018 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.018 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.018 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.018 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:452 3.1s0 3:8dfc6abc:::smithi17039300-28:head [append 0~425984] snapc 19=[19] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2023-10-20T03:35:47.018 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.840838+0000", 2023-10-20T03:35:47.019 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.0728175149999997, 2023-10-20T03:35:47.019 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.060832018000000002, 2023-10-20T03:35:47.019 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:47.019 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:47.019 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:47.019 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:47.020 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:47.020 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 452 2023-10-20T03:35:47.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.020 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:47.020 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.020 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:47.021 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.840838+0000", 2023-10-20T03:35:47.021 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.021 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:47.021 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.840838+0000", 2023-10-20T03:35:47.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.022 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.022 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.840841+0000", 2023-10-20T03:35:47.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4559999999999999e-06 2023-10-20T03:35:47.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.023 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.023 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843028+0000", 2023-10-20T03:35:47.023 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021872469999999998 2023-10-20T03:35:47.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.024 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.024 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.024 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843064+0000", 2023-10-20T03:35:47.024 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5686999999999999e-05 2023-10-20T03:35:47.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.025 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.025 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.025 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843086+0000", 2023-10-20T03:35:47.025 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1858999999999999e-05 2023-10-20T03:35:47.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.025 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.026 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.026 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.850898+0000", 2023-10-20T03:35:47.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0078129010000000006 2023-10-20T03:35:47.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.026 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.026 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:47.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.851240+0000", 2023-10-20T03:35:47.027 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000342085 2023-10-20T03:35:47.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:47.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.854106+0000", 2023-10-20T03:35:47.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002865271 2023-10-20T03:35:47.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:47.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.900943+0000", 2023-10-20T03:35:47.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.046837119000000003 2023-10-20T03:35:47.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.029 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:47.029 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.901551+0000", 2023-10-20T03:35:47.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00060791199999999995 2023-10-20T03:35:47.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.901670+0000", 2023-10-20T03:35:47.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119481 2023-10-20T03:35:47.030 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.030 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.030 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.031 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:453 3.1s0 3:8dfc6abc:::smithi17039300-28:head [append 0~786432] snapc 19=[19] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2023-10-20T03:35:47.031 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.857242+0000", 2023-10-20T03:35:47.031 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.0564137000000002, 2023-10-20T03:35:47.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.062724994000000006, 2023-10-20T03:35:47.032 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:47.032 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:47.032 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:47.032 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:47.032 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:47.032 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 453 2023-10-20T03:35:47.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.033 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:47.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:47.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.857242+0000", 2023-10-20T03:35:47.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.034 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:47.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.857242+0000", 2023-10-20T03:35:47.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.035 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.857245+0000", 2023-10-20T03:35:47.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2339999999999999e-06 2023-10-20T03:35:47.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.036 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.858828+0000", 2023-10-20T03:35:47.036 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0015830740000000001 2023-10-20T03:35:47.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.858877+0000", 2023-10-20T03:35:47.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8767000000000003e-05 2023-10-20T03:35:47.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.037 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.858899+0000", 2023-10-20T03:35:47.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.179e-05 2023-10-20T03:35:47.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.873979+0000", 2023-10-20T03:35:47.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01507966 2023-10-20T03:35:47.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:47.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.874252+0000", 2023-10-20T03:35:47.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00027352399999999998 2023-10-20T03:35:47.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.040 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.040 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:47.040 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.876430+0000", 2023-10-20T03:35:47.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021774950000000002 2023-10-20T03:35:47.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.040 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.041 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:47.041 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.901212+0000", 2023-10-20T03:35:47.041 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024782812000000001 2023-10-20T03:35:47.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.041 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.041 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:47.042 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.919823+0000", 2023-10-20T03:35:47.042 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018611043000000001 2023-10-20T03:35:47.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.042 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.042 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.042 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.919967+0000", 2023-10-20T03:35:47.043 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000143595 2023-10-20T03:35:47.043 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.043 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.043 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.043 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.044 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:454 3.1s0 3:8dfc6abc:::smithi17039300-28:head [append 0~376832] snapc 19=[19] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2023-10-20T03:35:47.044 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.864382+0000", 2023-10-20T03:35:47.044 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.0492738990000001, 2023-10-20T03:35:47.044 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.055923945000000003, 2023-10-20T03:35:47.044 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:47.045 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:47.045 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:47.045 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:47.045 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:47.045 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 454 2023-10-20T03:35:47.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.046 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:47.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.046 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:47.046 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.864382+0000", 2023-10-20T03:35:47.046 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.047 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:47.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.864382+0000", 2023-10-20T03:35:47.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.047 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.864390+0000", 2023-10-20T03:35:47.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6000000000000007e-06 2023-10-20T03:35:47.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.864898+0000", 2023-10-20T03:35:47.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00050757099999999996 2023-10-20T03:35:47.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.864927+0000", 2023-10-20T03:35:47.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8955e-05 2023-10-20T03:35:47.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.050 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.050 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.864941+0000", 2023-10-20T03:35:47.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4477e-05 2023-10-20T03:35:47.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.892069+0000", 2023-10-20T03:35:47.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027127277000000002 2023-10-20T03:35:47.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:47.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.892323+0000", 2023-10-20T03:35:47.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025378500000000002 2023-10-20T03:35:47.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:47.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.894008+0000", 2023-10-20T03:35:47.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0016850649999999999 2023-10-20T03:35:47.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:47.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.903147+0000", 2023-10-20T03:35:47.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0091392859999999999 2023-10-20T03:35:47.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:47.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.920181+0000", 2023-10-20T03:35:47.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017033716000000001 2023-10-20T03:35:47.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.920306+0000", 2023-10-20T03:35:47.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125213 2023-10-20T03:35:47.056 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.056 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.056 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.057 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:455 3.1s0 3:8dfc6abc:::smithi17039300-28:head [setxattr _header (57) in=7b] snapc 19=[19] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2023-10-20T03:35:47.057 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.864971+0000", 2023-10-20T03:35:47.057 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.0486850639999998, 2023-10-20T03:35:47.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.055934930000000001, 2023-10-20T03:35:47.057 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:47.058 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:47.058 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:47.058 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:47.058 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:47.058 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 455 2023-10-20T03:35:47.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.059 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:47.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:47.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.864971+0000", 2023-10-20T03:35:47.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:47.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.864971+0000", 2023-10-20T03:35:47.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.864972+0000", 2023-10-20T03:35:47.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.336e-06 2023-10-20T03:35:47.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.864980+0000", 2023-10-20T03:35:47.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.554e-06 2023-10-20T03:35:47.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.864997+0000", 2023-10-20T03:35:47.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7592999999999999e-05 2023-10-20T03:35:47.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.865010+0000", 2023-10-20T03:35:47.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2525e-05 2023-10-20T03:35:47.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.901782+0000", 2023-10-20T03:35:47.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.036772526999999999 2023-10-20T03:35:47.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:47.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.901950+0000", 2023-10-20T03:35:47.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000168145 2023-10-20T03:35:47.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:47.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.902574+0000", 2023-10-20T03:35:47.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000623822 2023-10-20T03:35:47.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:47.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.904524+0000", 2023-10-20T03:35:47.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001949922 2023-10-20T03:35:47.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:47.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.920729+0000", 2023-10-20T03:35:47.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016205265999999999 2023-10-20T03:35:47.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.920906+0000", 2023-10-20T03:35:47.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017624000000000001 2023-10-20T03:35:47.069 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.069 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.069 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.070 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:456 3.1s0 3:8dfc6abc:::smithi17039300-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2023-10-20T03:35:47.070 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.865033+0000", 2023-10-20T03:35:47.070 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.0486232219999998, 2023-10-20T03:35:47.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.062189318, 2023-10-20T03:35:47.071 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:47.071 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:35:47.071 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:47.071 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:47.071 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:47.071 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 456 2023-10-20T03:35:47.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.072 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:47.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:47.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.865033+0000", 2023-10-20T03:35:47.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:47.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.865033+0000", 2023-10-20T03:35:47.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.865034+0000", 2023-10-20T03:35:47.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1829999999999999e-06 2023-10-20T03:35:47.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.865039+0000", 2023-10-20T03:35:47.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7049999999999996e-06 2023-10-20T03:35:47.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.865057+0000", 2023-10-20T03:35:47.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.772e-05 2023-10-20T03:35:47.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.865070+0000", 2023-10-20T03:35:47.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2938999999999999e-05 2023-10-20T03:35:47.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.902979+0000", 2023-10-20T03:35:47.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.03790905 2023-10-20T03:35:47.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:47.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.903066+0000", 2023-10-20T03:35:47.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6786999999999994e-05 2023-10-20T03:35:47.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.920987+0000", 2023-10-20T03:35:47.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017920967999999999 2023-10-20T03:35:47.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:47.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.921158+0000", 2023-10-20T03:35:47.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000171082 2023-10-20T03:35:47.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.927222+0000", 2023-10-20T03:35:47.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0060638840000000003 2023-10-20T03:35:47.081 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.081 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.082 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.082 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:458 3.7s0 3:e9fa8fd1:::smithi17039300-38:head [append 0~458752] snapc 19=[19] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2023-10-20T03:35:47.082 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.884232+0000", 2023-10-20T03:35:47.082 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.0294241780000002, 2023-10-20T03:35:47.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043249271999999998, 2023-10-20T03:35:47.083 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:47.083 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:47.083 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:47.083 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:47.084 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:47.084 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 458 2023-10-20T03:35:47.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.084 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:47.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:47.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.884232+0000", 2023-10-20T03:35:47.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:47.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.884232+0000", 2023-10-20T03:35:47.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.884234+0000", 2023-10-20T03:35:47.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4779999999999998e-06 2023-10-20T03:35:47.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.884937+0000", 2023-10-20T03:35:47.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00070292399999999995 2023-10-20T03:35:47.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.884962+0000", 2023-10-20T03:35:47.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5477e-05 2023-10-20T03:35:47.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.884978+0000", 2023-10-20T03:35:47.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6002999999999999e-05 2023-10-20T03:35:47.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.885125+0000", 2023-10-20T03:35:47.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014684400000000001 2023-10-20T03:35:47.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:47.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.885323+0000", 2023-10-20T03:35:47.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019740799999999999 2023-10-20T03:35:47.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:47.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.887638+0000", 2023-10-20T03:35:47.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023149580000000002 2023-10-20T03:35:47.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:47.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.912721+0000", 2023-10-20T03:35:47.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025082882000000001 2023-10-20T03:35:47.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:47.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.927010+0000", 2023-10-20T03:35:47.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014289313999999999 2023-10-20T03:35:47.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.927481+0000", 2023-10-20T03:35:47.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00047098399999999998 2023-10-20T03:35:47.094 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.094 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.095 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.095 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:459 3.7s0 3:e9fa8fd1:::smithi17039300-38:head [append 0~688128] snapc 19=[19] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2023-10-20T03:35:47.095 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.893589+0000", 2023-10-20T03:35:47.095 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.0200667939999999, 2023-10-20T03:35:47.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.034616571999999998, 2023-10-20T03:35:47.096 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:47.096 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:47.096 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:47.096 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:47.096 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:47.097 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 459 2023-10-20T03:35:47.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.097 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:47.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:47.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.893589+0000", 2023-10-20T03:35:47.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:47.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.893589+0000", 2023-10-20T03:35:47.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.893591+0000", 2023-10-20T03:35:47.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.827e-06 2023-10-20T03:35:47.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.894420+0000", 2023-10-20T03:35:47.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00082913699999999997 2023-10-20T03:35:47.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.894452+0000", 2023-10-20T03:35:47.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2089999999999999e-05 2023-10-20T03:35:47.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.894469+0000", 2023-10-20T03:35:47.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6498999999999999e-05 2023-10-20T03:35:47.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.894993+0000", 2023-10-20T03:35:47.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000524341 2023-10-20T03:35:47.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:47.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.895181+0000", 2023-10-20T03:35:47.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018784699999999999 2023-10-20T03:35:47.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:47.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.896962+0000", 2023-10-20T03:35:47.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0017809080000000001 2023-10-20T03:35:47.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:47.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.926766+0000", 2023-10-20T03:35:47.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.029804359999999998 2023-10-20T03:35:47.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:47.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.927841+0000", 2023-10-20T03:35:47.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001075016 2023-10-20T03:35:47.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.928206+0000", 2023-10-20T03:35:47.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000364547 2023-10-20T03:35:47.107 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.107 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.107 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.108 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:539 3.1s0 3:8453ea10:::smithi17039300-9:head [append 0~606208] snapc 1e=[1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2023-10-20T03:35:47.108 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:45.885579+0000", 2023-10-20T03:35:47.108 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.0280763799999999, 2023-10-20T03:35:47.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.052141187999999998, 2023-10-20T03:35:47.108 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:47.109 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:47.109 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:47.109 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:47.109 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:47.109 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 539 2023-10-20T03:35:47.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.110 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:47.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:47.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.885579+0000", 2023-10-20T03:35:47.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:47.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.885579+0000", 2023-10-20T03:35:47.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.885583+0000", 2023-10-20T03:35:47.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.146e-06 2023-10-20T03:35:47.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.886598+0000", 2023-10-20T03:35:47.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0010150890000000001 2023-10-20T03:35:47.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.886637+0000", 2023-10-20T03:35:47.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9597000000000001e-05 2023-10-20T03:35:47.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.886662+0000", 2023-10-20T03:35:47.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5185999999999999e-05 2023-10-20T03:35:47.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.886799+0000", 2023-10-20T03:35:47.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136274 2023-10-20T03:35:47.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op must wait for map", 2023-10-20T03:35:47.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.886810+0000", 2023-10-20T03:35:47.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1454000000000001e-05 2023-10-20T03:35:47.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.893571+0000", 2023-10-20T03:35:47.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0067606230000000003 2023-10-20T03:35:47.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:47.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.893847+0000", 2023-10-20T03:35:47.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00027603799999999999 2023-10-20T03:35:47.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:47.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.896771+0000", 2023-10-20T03:35:47.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0029244280000000002 2023-10-20T03:35:47.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:47.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.924351+0000", 2023-10-20T03:35:47.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027580111000000001 2023-10-20T03:35:47.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:47.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.937439+0000", 2023-10-20T03:35:47.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013087368 2023-10-20T03:35:47.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.937721+0000", 2023-10-20T03:35:47.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028187399999999999 2023-10-20T03:35:47.122 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.122 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.122 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.122 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:540 3.1s0 3:8453ea10:::smithi17039300-9:head [append 0~688128] snapc 1e=[1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2023-10-20T03:35:47.123 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:45.895954+0000", 2023-10-20T03:35:47.123 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.0177017589999999, 2023-10-20T03:35:47.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.046294450000000001, 2023-10-20T03:35:47.123 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:47.123 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:47.124 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:47.124 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:47.124 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:47.124 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 540 2023-10-20T03:35:47.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.125 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:47.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:47.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.895954+0000", 2023-10-20T03:35:47.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:47.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.895954+0000", 2023-10-20T03:35:47.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.895957+0000", 2023-10-20T03:35:47.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9069999999999999e-06 2023-10-20T03:35:47.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.897147+0000", 2023-10-20T03:35:47.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0011897310000000001 2023-10-20T03:35:47.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.897179+0000", 2023-10-20T03:35:47.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2286000000000002e-05 2023-10-20T03:35:47.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.897202+0000", 2023-10-20T03:35:47.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2687000000000001e-05 2023-10-20T03:35:47.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.130 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.909307+0000", 2023-10-20T03:35:47.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012105757999999999 2023-10-20T03:35:47.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:47.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.909631+0000", 2023-10-20T03:35:47.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032408799999999999 2023-10-20T03:35:47.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:47.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.911906+0000", 2023-10-20T03:35:47.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022745859999999999 2023-10-20T03:35:47.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:47.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.936788+0000", 2023-10-20T03:35:47.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024881983999999999 2023-10-20T03:35:47.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:47.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.942120+0000", 2023-10-20T03:35:47.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053315070000000001 2023-10-20T03:35:47.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.134 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.942248+0000", 2023-10-20T03:35:47.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000128916 2023-10-20T03:35:47.135 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.135 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.135 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.135 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:541 3.1s0 3:8453ea10:::smithi17039300-9:head [append 0~475136] snapc 1e=[1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2023-10-20T03:35:47.135 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:45.904085+0000", 2023-10-20T03:35:47.136 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.009571247, 2023-10-20T03:35:47.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039166382999999999, 2023-10-20T03:35:47.136 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:47.136 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:47.136 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:47.136 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:47.137 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:47.137 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 541 2023-10-20T03:35:47.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.137 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:47.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:47.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.904085+0000", 2023-10-20T03:35:47.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:47.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.904085+0000", 2023-10-20T03:35:47.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.139 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.139 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.904087+0000", 2023-10-20T03:35:47.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4430000000000002e-06 2023-10-20T03:35:47.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.904886+0000", 2023-10-20T03:35:47.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00079899099999999998 2023-10-20T03:35:47.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.904925+0000", 2023-10-20T03:35:47.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9350000000000001e-05 2023-10-20T03:35:47.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.904947+0000", 2023-10-20T03:35:47.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1199000000000002e-05 2023-10-20T03:35:47.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.143 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.924506+0000", 2023-10-20T03:35:47.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019559778999999999 2023-10-20T03:35:47.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:47.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.924690+0000", 2023-10-20T03:35:47.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018407499999999999 2023-10-20T03:35:47.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:47.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.926247+0000", 2023-10-20T03:35:47.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001556158 2023-10-20T03:35:47.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:47.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.937036+0000", 2023-10-20T03:35:47.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010788979000000001 2023-10-20T03:35:47.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:47.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.943118+0000", 2023-10-20T03:35:47.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0060828669999999996 2023-10-20T03:35:47.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.147 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.147 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.943251+0000", 2023-10-20T03:35:47.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013254200000000001 2023-10-20T03:35:47.147 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.148 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.148 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.148 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:542 3.1s0 3:8453ea10:::smithi17039300-9:head [setxattr _header (56) in=7b] snapc 1e=[1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2023-10-20T03:35:47.148 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:45.904990+0000", 2023-10-20T03:35:47.149 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.0086658239999999, 2023-10-20T03:35:47.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039484561000000001, 2023-10-20T03:35:47.149 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:47.149 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:47.149 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:47.149 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:47.150 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:47.150 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 542 2023-10-20T03:35:47.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.150 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:47.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:47.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.904990+0000", 2023-10-20T03:35:47.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:47.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.904990+0000", 2023-10-20T03:35:47.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.904991+0000", 2023-10-20T03:35:47.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4529999999999999e-06 2023-10-20T03:35:47.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.905000+0000", 2023-10-20T03:35:47.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3000000000000002e-06 2023-10-20T03:35:47.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.154 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.905026+0000", 2023-10-20T03:35:47.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5947000000000001e-05 2023-10-20T03:35:47.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.905043+0000", 2023-10-20T03:35:47.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7796e-05 2023-10-20T03:35:47.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.935045+0000", 2023-10-20T03:35:47.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030001843 2023-10-20T03:35:47.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:47.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.935233+0000", 2023-10-20T03:35:47.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018796199999999999 2023-10-20T03:35:47.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:35:47.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.936103+0000", 2023-10-20T03:35:47.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00086985000000000003 2023-10-20T03:35:47.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:35:47.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.938381+0000", 2023-10-20T03:35:47.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022779530000000001 2023-10-20T03:35:47.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:35:47.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.944328+0000", 2023-10-20T03:35:47.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0059469709999999997 2023-10-20T03:35:47.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.944475+0000", 2023-10-20T03:35:47.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014648600000000001 2023-10-20T03:35:47.161 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.161 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.161 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.161 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:543 3.1s0 3:8453ea10:::smithi17039300-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2023-10-20T03:35:47.162 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:45.905068+0000", 2023-10-20T03:35:47.162 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.0085873409999999, 2023-10-20T03:35:47.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043167012999999997, 2023-10-20T03:35:47.162 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:35:47.162 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:35:47.162 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:35:47.163 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:35:47.163 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:47.163 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 543 2023-10-20T03:35:47.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.163 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:35:47.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:35:47.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.905068+0000", 2023-10-20T03:35:47.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:35:47.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.905068+0000", 2023-10-20T03:35:47.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:35:47.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:35:47.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.905070+0000", 2023-10-20T03:35:47.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3200000000000001e-06 2023-10-20T03:35:47.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:35:47.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.905076+0000", 2023-10-20T03:35:47.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7529999999999998e-06 2023-10-20T03:35:47.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:35:47.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.905098+0000", 2023-10-20T03:35:47.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2782999999999999e-05 2023-10-20T03:35:47.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:35:47.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.905116+0000", 2023-10-20T03:35:47.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7229999999999999e-05 2023-10-20T03:35:47.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.938816+0000", 2023-10-20T03:35:47.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033700238 2023-10-20T03:35:47.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.170 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:47.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.939014+0000", 2023-10-20T03:35:47.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019848199999999999 2023-10-20T03:35:47.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.170 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:35:47.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.945342+0000", 2023-10-20T03:35:47.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0063274220000000001 2023-10-20T03:35:47.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:35:47.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.945640+0000", 2023-10-20T03:35:47.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000298394 2023-10-20T03:35:47.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:35:47.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:35:47.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:35:47.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:45.948235+0000", 2023-10-20T03:35:47.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025953909999999998 2023-10-20T03:35:47.173 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.173 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.173 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.173 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:35:47.173 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:35:47.173 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:35:47.175 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-10-20T03:35:47.175 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:47.174+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:47.276 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:47.272+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:47.303 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:35:47.310 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:47.310 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2023-10-20T03:35:47.311 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:47.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-10-20T03:35:47.376 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:47.373+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:47.444 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:47.444 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2023-10-20T03:35:47.445 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2023-10-20T03:35:47.445 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:47.454 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:47.455 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.460 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-10-20T03:35:47.474 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:47.475 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.477 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:47.472+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:47.492 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:47.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.513 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:47.513 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.533 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:47.534 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.578 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:47.576+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:47.586 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:47.587 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:47.601 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-10-20T03:35:47.680 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:47.678+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:47.727 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:47.728 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:47.728 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2023-10-20T03:35:47.728 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:47.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-10-20T03:35:47.761 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:47.761 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.762 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:47.780 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:47.778+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:47.784 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:47.784 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.803 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:47.804 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.825 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:47.825 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.845 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:47.845 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.866 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:47.866 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.871 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:47.871 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:47.877 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2023-10-20T03:35:47.881 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:47.877+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:47.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-10-20T03:35:47.887 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:47.888 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.888 INFO:tasks.rados.rados.0.smithi170.stdout:226: done (3 left) 2023-10-20T03:35:47.889 INFO:tasks.rados.rados.0.smithi170.stdout:227: snap_create 2023-10-20T03:35:47.889 INFO:tasks.rados.rados.0.smithi170.stdout:223: finishing write tid 1 to smithi17039300-9 2023-10-20T03:35:47.889 INFO:tasks.rados.rados.0.smithi170.stdout:223: finishing write tid 2 to smithi17039300-9 2023-10-20T03:35:47.889 INFO:tasks.rados.rados.0.smithi170.stdout:223: finishing write tid 3 to smithi17039300-9 2023-10-20T03:35:47.889 INFO:tasks.rados.rados.0.smithi170.stdout:223: finishing write tid 4 to smithi17039300-9 2023-10-20T03:35:47.889 INFO:tasks.rados.rados.0.smithi170.stdout:223: finishing write tid 5 to smithi17039300-9 2023-10-20T03:35:47.890 INFO:tasks.rados.rados.0.smithi170.stdout:223: oid 9 updating version 0 to 64 2023-10-20T03:35:47.890 INFO:tasks.rados.rados.0.smithi170.stdout:223: oid 9 updating version 64 to 65 2023-10-20T03:35:47.890 INFO:tasks.rados.rados.0.smithi170.stdout:223: oid 9 updating version 65 to 66 2023-10-20T03:35:47.890 INFO:tasks.rados.rados.0.smithi170.stdout:223: oid 9 version 66 is already newer than 63 2023-10-20T03:35:47.890 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 9 v 66 (ObjNum 105 snap 16 seq_num 105) dirty exists 2023-10-20T03:35:47.891 INFO:tasks.rados.rados.0.smithi170.stdout:223: left oid 9 (ObjNum 105 snap 16 seq_num 105) 2023-10-20T03:35:47.891 INFO:tasks.rados.rados.0.smithi170.stdout:225: expect (ObjNum 104 snap 13 seq_num 104) 2023-10-20T03:35:47.911 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:47.911 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.930 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:35:47.929+0000 7f145659c640 0 -- 172.21.15.33:0/3012380715 >> [v2:172.21.15.64:6816/2940007286,v1:172.21.15.64:6817/2940007286] conn(0x7f145821efe0 msgr2=0x7f14582213d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-10-20T03:35:47.932 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:47.955 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:47.955 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:35:47.959 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:47.959 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:47.981 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:47.978+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:48.013 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:48.013 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:48.027 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-10-20T03:35:48.082 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:48.081+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:48.145 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:48.145 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:48.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-10-20T03:35:48.183 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:48.182+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:48.284 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:48.281+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:48.291 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:48.291 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:48.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-10-20T03:35:48.385 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:48.380+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:48.433 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:48.433 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:48.447 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-10-20T03:35:48.485 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:48.484+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:48.574 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:48.575 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:48.586 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:48.582+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:48.589 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-10-20T03:35:48.687 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:48.685+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:48.754 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:48.755 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:48.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-10-20T03:35:48.787 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:48.784+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:48.884 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:48.885 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:48.888 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:48.885+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:48.898 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-10-20T03:35:48.907 INFO:tasks.rados.rados.0.smithi170.stdout:222: done (3 left) 2023-10-20T03:35:48.907 INFO:tasks.rados.rados.0.smithi170.stdout:223: done (2 left) 2023-10-20T03:35:48.908 INFO:tasks.rados.rados.0.smithi170.stdout:225: done (1 left) 2023-10-20T03:35:48.908 INFO:tasks.rados.rados.0.smithi170.stdout:227: done (0 left) 2023-10-20T03:35:48.908 INFO:tasks.rados.rados.0.smithi170.stdout:228: snap_remove snap 15 2023-10-20T03:35:48.989 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:48.985+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:49.018 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:49.019 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:49.033 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-10-20T03:35:49.090 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:49.088+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:49.191 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:49.188+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:49.200 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:49.200 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:49.214 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-10-20T03:35:49.292 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:49.289+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:49.330 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:49.330 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:49.343 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-10-20T03:35:49.393 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:49.390+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:49.462 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:49.463 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:49.478 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-10-20T03:35:49.494 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:49.492+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:49.595 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:49.593+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:49.641 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:49.641 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:49.655 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-10-20T03:35:49.696 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:49.693+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:49.771 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:49.771 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:49.785 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-10-20T03:35:49.797 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:49.793+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:49.898 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:49.894+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:49.900 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-10-20T03:35:49.907 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:49.908 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:49.921 INFO:tasks.rados.rados.0.smithi170.stdout:228: done (0 left) 2023-10-20T03:35:49.921 INFO:tasks.rados.rados.0.smithi170.stdout:229: snap_create 2023-10-20T03:35:49.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-10-20T03:35:49.960 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:35:49.999 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:49.996+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:50.088 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:50.089 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:50.100 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:50.097+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:50.102 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-10-20T03:35:50.113 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:50.113 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.133 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:50.133 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.156 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:50.156 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.172 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:50.173 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.190 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:50.191 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.202 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:50.200+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:50.210 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:50.211 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.211 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:50.226 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:50.226 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:50.228 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:50.228 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.241 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-10-20T03:35:50.246 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:50.246 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.270 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:50.270 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.293 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:50.293 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.303 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:50.302+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:50.312 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:50.313 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.331 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:50.331 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.353 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:50.353 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.367 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:50.367 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:50.367 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:50.368 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:50.375 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:50.375 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.376 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:35:50.373+0000 7fc4667fc640 0 -- 172.21.15.33:0/3454916435 >> [v2:172.21.15.170:6819/826332016,v1:172.21.15.170:6822/826332016] conn(0x7fc468221470 msgr2=0x7fc468223860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-10-20T03:35:50.382 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-10-20T03:35:50.426 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:50.425+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:50.505 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:50.504+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:50.555 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:50.555 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:50.555 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:35:50.556 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:35:50.556 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:50.569 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-10-20T03:35:50.598 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:50.599 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:35:50.602 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:50.602 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:50.606 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:50.604+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:50.698 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:50.699 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:50.699 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:50.699 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:35:50.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.700 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s2 36/31 ECSubWrite(tid=21, reqid=client.4570.0:198, at_version=36'13, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:50.700 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.265066+0000", 2023-10-20T03:35:50.700 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 32.430419716000003, 2023-10-20T03:35:50.700 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.007859797, 2023-10-20T03:35:50.701 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.701 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.701 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.702 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.702 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265066+0000", 2023-10-20T03:35:50.702 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.702 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.703 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.703 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265066+0000", 2023-10-20T03:35:50.703 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.704 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.704 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265066+0000", 2023-10-20T03:35:50.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.35e-07 2023-10-20T03:35:50.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.705 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.705 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265361+0000", 2023-10-20T03:35:50.705 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00029424799999999999 2023-10-20T03:35:50.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.707 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.707 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265372+0000", 2023-10-20T03:35:50.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1834000000000001e-05 2023-10-20T03:35:50.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265382+0000", 2023-10-20T03:35:50.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4059999999999994e-06 2023-10-20T03:35:50.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.709 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.709 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265460+0000", 2023-10-20T03:35:50.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7844999999999998e-05 2023-10-20T03:35:50.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.712 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.712 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265467+0000", 2023-10-20T03:35:50.712 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7530000000000001e-06 2023-10-20T03:35:50.712 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.712 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.713 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.272862+0000", 2023-10-20T03:35:50.714 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073944290000000001 2023-10-20T03:35:50.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.715 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.272926+0000", 2023-10-20T03:35:50.715 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4146999999999994e-05 2023-10-20T03:35:50.715 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.716 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.716 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.716 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.716 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:206 3.fs0 3:f289eea4:::smithi17039300-43:head [stat,delete] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-10-20T03:35:50.717 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.281026+0000", 2023-10-20T03:35:50.717 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 32.414460005000002, 2023-10-20T03:35:50.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081465319999999997, 2023-10-20T03:35:50.717 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.718 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:50.718 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:50.718 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:50.718 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:50.718 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 206 2023-10-20T03:35:50.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.719 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.719 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.720 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281026+0000", 2023-10-20T03:35:50.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281026+0000", 2023-10-20T03:35:50.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.722 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.722 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281028+0000", 2023-10-20T03:35:50.722 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.73e-06 2023-10-20T03:35:50.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.723 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.723 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.723 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281034+0000", 2023-10-20T03:35:50.723 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7499999999999997e-06 2023-10-20T03:35:50.724 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.724 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.724 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.725 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281054+0000", 2023-10-20T03:35:50.725 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9601000000000001e-05 2023-10-20T03:35:50.725 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.725 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.725 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.726 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281066+0000", 2023-10-20T03:35:50.726 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1538e-05 2023-10-20T03:35:50.727 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.727 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.727 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.727 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281120+0000", 2023-10-20T03:35:50.727 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4268999999999998e-05 2023-10-20T03:35:50.728 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.728 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.728 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:50.729 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281193+0000", 2023-10-20T03:35:50.729 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2675999999999993e-05 2023-10-20T03:35:50.730 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.730 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.731 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.731 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.281910+0000", 2023-10-20T03:35:50.731 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00071703499999999998 2023-10-20T03:35:50.732 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.732 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.732 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.732 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.283019+0000", 2023-10-20T03:35:50.733 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011095860000000001 2023-10-20T03:35:50.733 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.733 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.734 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:50.734 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.289127+0000", 2023-10-20T03:35:50.735 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00610796 2023-10-20T03:35:50.735 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.735 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.735 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.736 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.289172+0000", 2023-10-20T03:35:50.736 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5386999999999998e-05 2023-10-20T03:35:50.736 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.736 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.736 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.737 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.737 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.737 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s5 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:35:50.737 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.356693+0000", 2023-10-20T03:35:50.738 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.338792569999999, 2023-10-20T03:35:50.738 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010267518, 2023-10-20T03:35:50.738 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.738 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.739 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.739 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.739 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.739 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356693+0000", 2023-10-20T03:35:50.740 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.740 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.740 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.740 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.741 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356693+0000", 2023-10-20T03:35:50.741 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.741 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.742 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.742 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.742 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356694+0000", 2023-10-20T03:35:50.742 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5000000000000002e-07 2023-10-20T03:35:50.742 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.743 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.743 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.743 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357174+0000", 2023-10-20T03:35:50.743 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047994599999999998 2023-10-20T03:35:50.744 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.744 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.744 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.744 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357194+0000", 2023-10-20T03:35:50.745 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.978e-05 2023-10-20T03:35:50.745 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.745 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.745 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.746 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357209+0000", 2023-10-20T03:35:50.746 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5506e-05 2023-10-20T03:35:50.746 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.746 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.747 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.747 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357336+0000", 2023-10-20T03:35:50.747 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012640000000000001 2023-10-20T03:35:50.747 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.748 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.748 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.748 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357353+0000", 2023-10-20T03:35:50.748 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6867e-05 2023-10-20T03:35:50.748 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.749 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.749 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.749 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.366907+0000", 2023-10-20T03:35:50.749 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.009554412 2023-10-20T03:35:50.749 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.750 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.750 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.750 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.366961+0000", 2023-10-20T03:35:50.750 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3956999999999999e-05 2023-10-20T03:35:50.750 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.751 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.751 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.751 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.751 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.752 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:330 3.cs0 3:3382ea17:::smithi17039300-35:head [append 0~0] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:50.752 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.371102+0000", 2023-10-20T03:35:50.752 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.324383812000001, 2023-10-20T03:35:50.752 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0080082599999999997, 2023-10-20T03:35:50.752 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.753 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:50.753 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:50.753 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:50.753 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:50.754 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 330 2023-10-20T03:35:50.754 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.754 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.754 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.755 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.755 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371102+0000", 2023-10-20T03:35:50.755 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.755 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.756 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.756 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.756 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371102+0000", 2023-10-20T03:35:50.756 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.757 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.757 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.757 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.757 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371104+0000", 2023-10-20T03:35:50.757 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6270000000000001e-06 2023-10-20T03:35:50.758 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.758 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.758 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.758 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371126+0000", 2023-10-20T03:35:50.759 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2566999999999999e-05 2023-10-20T03:35:50.759 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.759 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.759 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.759 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371159+0000", 2023-10-20T03:35:50.760 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2255999999999997e-05 2023-10-20T03:35:50.760 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.760 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.760 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.760 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371173+0000", 2023-10-20T03:35:50.761 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4609e-05 2023-10-20T03:35:50.761 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.761 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.761 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.761 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371246+0000", 2023-10-20T03:35:50.762 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3030999999999999e-05 2023-10-20T03:35:50.762 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.762 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.762 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:50.762 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371343+0000", 2023-10-20T03:35:50.763 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7163000000000005e-05 2023-10-20T03:35:50.763 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.763 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.763 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.763 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.372056+0000", 2023-10-20T03:35:50.764 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00071231599999999995 2023-10-20T03:35:50.764 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.764 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.764 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.765 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.374405+0000", 2023-10-20T03:35:50.765 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0023489299999999999 2023-10-20T03:35:50.765 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.765 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.766 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:50.766 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.379046+0000", 2023-10-20T03:35:50.766 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0046410230000000002 2023-10-20T03:35:50.766 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.767 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.767 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.767 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.379110+0000", 2023-10-20T03:35:50.767 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4738000000000003e-05 2023-10-20T03:35:50.767 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.768 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.768 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.768 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.768 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.768 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:331 3.cs0 3:3382ea17:::smithi17039300-35:head [rollback a] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:50.769 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.371212+0000", 2023-10-20T03:35:50.769 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.324274184, 2023-10-20T03:35:50.769 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010542586, 2023-10-20T03:35:50.769 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.770 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:50.770 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:50.770 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:50.770 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:50.770 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 331 2023-10-20T03:35:50.771 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.771 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.771 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.771 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.772 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371212+0000", 2023-10-20T03:35:50.772 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.772 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.772 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.773 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.773 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371212+0000", 2023-10-20T03:35:50.773 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.773 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.773 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.774 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.774 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371213+0000", 2023-10-20T03:35:50.774 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1999999999999998e-07 2023-10-20T03:35:50.774 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.775 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.775 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.775 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371219+0000", 2023-10-20T03:35:50.775 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2600000000000002e-06 2023-10-20T03:35:50.775 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.776 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.776 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.776 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371233+0000", 2023-10-20T03:35:50.776 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3816e-05 2023-10-20T03:35:50.777 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.777 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.777 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.777 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371241+0000", 2023-10-20T03:35:50.777 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5569999999999996e-06 2023-10-20T03:35:50.778 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.778 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.778 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.778 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.372719+0000", 2023-10-20T03:35:50.779 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014780539999999999 2023-10-20T03:35:50.779 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.779 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.779 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:50.779 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.372777+0000", 2023-10-20T03:35:50.780 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7485999999999998e-05 2023-10-20T03:35:50.780 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.780 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.780 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.780 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.373138+0000", 2023-10-20T03:35:50.781 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000361679 2023-10-20T03:35:50.781 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.781 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.781 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.781 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.376244+0000", 2023-10-20T03:35:50.782 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003105652 2023-10-20T03:35:50.782 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.782 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.782 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:50.782 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381631+0000", 2023-10-20T03:35:50.783 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053871179999999998 2023-10-20T03:35:50.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.783 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.783 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.783 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381754+0000", 2023-10-20T03:35:50.784 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000123144 2023-10-20T03:35:50.784 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.784 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.785 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.785 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:332 3.cs0 3:3382ea17:::smithi17039300-35:head [append 0~0] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:50.785 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.371334+0000", 2023-10-20T03:35:50.786 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.324151467, 2023-10-20T03:35:50.786 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010711231, 2023-10-20T03:35:50.786 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.787 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:50.787 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:50.787 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:50.787 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:50.787 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 332 2023-10-20T03:35:50.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.788 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.788 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.788 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.788 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371334+0000", 2023-10-20T03:35:50.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.789 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.790 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371334+0000", 2023-10-20T03:35:50.790 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.790 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.790 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.791 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371335+0000", 2023-10-20T03:35:50.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.47e-07 2023-10-20T03:35:50.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.792 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.792 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371340+0000", 2023-10-20T03:35:50.792 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5800000000000002e-06 2023-10-20T03:35:50.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.793 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.793 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371366+0000", 2023-10-20T03:35:50.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5639e-05 2023-10-20T03:35:50.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371379+0000", 2023-10-20T03:35:50.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.346e-05 2023-10-20T03:35:50.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.373793+0000", 2023-10-20T03:35:50.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024143049999999998 2023-10-20T03:35:50.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.797 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:50.797 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.373855+0000", 2023-10-20T03:35:50.797 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1277000000000003e-05 2023-10-20T03:35:50.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.798 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.798 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.374199+0000", 2023-10-20T03:35:50.798 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00034445500000000002 2023-10-20T03:35:50.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.802 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.802 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.803 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.376260+0000", 2023-10-20T03:35:50.803 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020611050000000001 2023-10-20T03:35:50.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.804 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:50.804 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381945+0000", 2023-10-20T03:35:50.804 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056847690000000001 2023-10-20T03:35:50.804 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.805 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.382046+0000", 2023-10-20T03:35:50.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000100794 2023-10-20T03:35:50.805 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.806 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.806 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.807 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:50.807 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.387979+0000", 2023-10-20T03:35:50.807 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.307507226999999, 2023-10-20T03:35:50.808 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018548599999999998, 2023-10-20T03:35:50.808 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.808 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.808 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.809 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.809 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387979+0000", 2023-10-20T03:35:50.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387979+0000", 2023-10-20T03:35:50.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387979+0000", 2023-10-20T03:35:50.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6000000000000003e-07 2023-10-20T03:35:50.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388153+0000", 2023-10-20T03:35:50.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017364600000000001 2023-10-20T03:35:50.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388173+0000", 2023-10-20T03:35:50.814 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9961000000000001e-05 2023-10-20T03:35:50.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.814 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.815 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388186+0000", 2023-10-20T03:35:50.815 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2403e-05 2023-10-20T03:35:50.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.815 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.815 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.816 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388293+0000", 2023-10-20T03:35:50.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010739300000000001 2023-10-20T03:35:50.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.816 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.817 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388307+0000", 2023-10-20T03:35:50.817 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3773e-05 2023-10-20T03:35:50.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.818 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.818 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406469+0000", 2023-10-20T03:35:50.818 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018162098000000002 2023-10-20T03:35:50.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.819 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.819 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.819 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406527+0000", 2023-10-20T03:35:50.819 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8566000000000001e-05 2023-10-20T03:35:50.820 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.820 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.820 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.820 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.821 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:50.821 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.394884+0000", 2023-10-20T03:35:50.821 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.300602129000001, 2023-10-20T03:35:50.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011674960999999999, 2023-10-20T03:35:50.822 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.822 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.822 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.823 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.823 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394884+0000", 2023-10-20T03:35:50.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.823 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.824 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.824 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394884+0000", 2023-10-20T03:35:50.824 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.825 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.825 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394886+0000", 2023-10-20T03:35:50.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0269999999999998e-06 2023-10-20T03:35:50.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.826 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.826 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395153+0000", 2023-10-20T03:35:50.826 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026702300000000002 2023-10-20T03:35:50.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.827 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.827 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.827 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395187+0000", 2023-10-20T03:35:50.827 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3738999999999997e-05 2023-10-20T03:35:50.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.828 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.828 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.828 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395211+0000", 2023-10-20T03:35:50.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4859e-05 2023-10-20T03:35:50.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395301+0000", 2023-10-20T03:35:50.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9553000000000001e-05 2023-10-20T03:35:50.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395310+0000", 2023-10-20T03:35:50.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9439999999999994e-06 2023-10-20T03:35:50.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.832 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.832 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.832 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406535+0000", 2023-10-20T03:35:50.832 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011225153999999999 2023-10-20T03:35:50.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.833 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.833 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406559+0000", 2023-10-20T03:35:50.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3662000000000001e-05 2023-10-20T03:35:50.834 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.834 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.834 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.835 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs0 48/47 ECSubWrite(tid=115, reqid=osd.13.0:115, at_version=48'27, trim_to=0'0, roll_forward_to=0'0))", 2023-10-20T03:35:50.835 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.389413+0000", 2023-10-20T03:35:50.835 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.306072666999999, 2023-10-20T03:35:50.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0097928949999999994, 2023-10-20T03:35:50.836 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.836 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.836 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.837 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.837 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389413+0000", 2023-10-20T03:35:50.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389413+0000", 2023-10-20T03:35:50.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.839 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.840 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389416+0000", 2023-10-20T03:35:50.840 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2170000000000001e-06 2023-10-20T03:35:50.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.840 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389474+0000", 2023-10-20T03:35:50.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8931999999999998e-05 2023-10-20T03:35:50.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389502+0000", 2023-10-20T03:35:50.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7538999999999998e-05 2023-10-20T03:35:50.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.843 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.843 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389531+0000", 2023-10-20T03:35:50.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9172000000000001e-05 2023-10-20T03:35:50.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.844 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389702+0000", 2023-10-20T03:35:50.845 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017064000000000001 2023-10-20T03:35:50.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.845 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.845 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389767+0000", 2023-10-20T03:35:50.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5297999999999997e-05 2023-10-20T03:35:50.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.399101+0000", 2023-10-20T03:35:50.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093336600000000006 2023-10-20T03:35:50.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.399206+0000", 2023-10-20T03:35:50.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000105437 2023-10-20T03:35:50.848 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.848 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.848 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.849 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:35:50.850 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.392018+0000", 2023-10-20T03:35:50.850 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.303468188, 2023-10-20T03:35:50.850 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037305552999999998, 2023-10-20T03:35:50.850 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.851 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.851 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392018+0000", 2023-10-20T03:35:50.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.853 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392018+0000", 2023-10-20T03:35:50.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.854 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.854 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392020+0000", 2023-10-20T03:35:50.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2029999999999999e-06 2023-10-20T03:35:50.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.854 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.855 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.855 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392961+0000", 2023-10-20T03:35:50.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00094143000000000004 2023-10-20T03:35:50.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.856 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.856 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392993+0000", 2023-10-20T03:35:50.856 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1795999999999998e-05 2023-10-20T03:35:50.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393083+0000", 2023-10-20T03:35:50.857 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9584000000000002e-05 2023-10-20T03:35:50.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393389+0000", 2023-10-20T03:35:50.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00030593699999999999 2023-10-20T03:35:50.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.859 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.859 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393416+0000", 2023-10-20T03:35:50.859 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7639000000000001e-05 2023-10-20T03:35:50.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429229+0000", 2023-10-20T03:35:50.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.035812384000000003 2023-10-20T03:35:50.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.861 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.861 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429323+0000", 2023-10-20T03:35:50.861 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4580000000000006e-05 2023-10-20T03:35:50.862 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.862 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.862 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.863 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.fs0 48/47 ECSubWrite(tid=35, reqid=osd.3.0:34, at_version=0'0, trim_to=0'0, roll_forward_to=48'21))", 2023-10-20T03:35:50.863 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.393341+0000", 2023-10-20T03:35:50.863 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.302144470999998, 2023-10-20T03:35:50.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092276119999999996, 2023-10-20T03:35:50.864 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.864 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.864 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.864 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393341+0000", 2023-10-20T03:35:50.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393341+0000", 2023-10-20T03:35:50.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393343+0000", 2023-10-20T03:35:50.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5740000000000001e-06 2023-10-20T03:35:50.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393374+0000", 2023-10-20T03:35:50.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0639000000000003e-05 2023-10-20T03:35:50.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393404+0000", 2023-10-20T03:35:50.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0521999999999997e-05 2023-10-20T03:35:50.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393425+0000", 2023-10-20T03:35:50.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0993e-05 2023-10-20T03:35:50.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393584+0000", 2023-10-20T03:35:50.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015895500000000001 2023-10-20T03:35:50.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393611+0000", 2023-10-20T03:35:50.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6361999999999999e-05 2023-10-20T03:35:50.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.402462+0000", 2023-10-20T03:35:50.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0088519600000000007 2023-10-20T03:35:50.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.402569+0000", 2023-10-20T03:35:50.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000106607 2023-10-20T03:35:50.878 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.878 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.879 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.879 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 49/31 ECSubWrite(tid=62, reqid=osd.12.0:62, at_version=49'25, trim_to=0'0, roll_forward_to=48'23))", 2023-10-20T03:35:50.880 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.407550+0000", 2023-10-20T03:35:50.880 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 19.287935585, 2023-10-20T03:35:50.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0069338179999999996, 2023-10-20T03:35:50.881 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.881 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.881 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.881 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.882 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407550+0000", 2023-10-20T03:35:50.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407550+0000", 2023-10-20T03:35:50.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407551+0000", 2023-10-20T03:35:50.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.27e-07 2023-10-20T03:35:50.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407579+0000", 2023-10-20T03:35:50.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8209999999999999e-05 2023-10-20T03:35:50.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.886 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.886 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407597+0000", 2023-10-20T03:35:50.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8365000000000001e-05 2023-10-20T03:35:50.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407609+0000", 2023-10-20T03:35:50.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2096000000000001e-05 2023-10-20T03:35:50.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407714+0000", 2023-10-20T03:35:50.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000104737 2023-10-20T03:35:50.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407728+0000", 2023-10-20T03:35:50.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3764000000000001e-05 2023-10-20T03:35:50.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.891 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.891 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414436+0000", 2023-10-20T03:35:50.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0067080270000000001 2023-10-20T03:35:50.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414484+0000", 2023-10-20T03:35:50.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8291999999999998e-05 2023-10-20T03:35:50.893 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.894 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.894 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.895 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 49/31 ECSubWrite(tid=63, reqid=osd.12.0:63, at_version=49'27, trim_to=0'0, roll_forward_to=48'23))", 2023-10-20T03:35:50.895 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.407639+0000", 2023-10-20T03:35:50.895 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 19.287847402000001, 2023-10-20T03:35:50.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0082579339999999998, 2023-10-20T03:35:50.896 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.896 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.897 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.897 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.897 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407639+0000", 2023-10-20T03:35:50.897 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.898 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.898 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.898 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.899 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407639+0000", 2023-10-20T03:35:50.899 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.900 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.900 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407639+0000", 2023-10-20T03:35:50.900 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4799999999999998e-07 2023-10-20T03:35:50.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.901 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.901 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.902 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407669+0000", 2023-10-20T03:35:50.902 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0235000000000001e-05 2023-10-20T03:35:50.902 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.903 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.903 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.903 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407681+0000", 2023-10-20T03:35:50.903 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2129e-05 2023-10-20T03:35:50.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.904 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.904 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.904 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407710+0000", 2023-10-20T03:35:50.904 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8816999999999999e-05 2023-10-20T03:35:50.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.905 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.905 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.906 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409705+0000", 2023-10-20T03:35:50.906 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019947340000000002 2023-10-20T03:35:50.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.907 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.907 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409718+0000", 2023-10-20T03:35:50.907 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2618e-05 2023-10-20T03:35:50.908 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.908 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.908 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.908 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415856+0000", 2023-10-20T03:35:50.909 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0061379379999999999 2023-10-20T03:35:50.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.909 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.910 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.910 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415896+0000", 2023-10-20T03:35:50.910 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0914999999999997e-05 2023-10-20T03:35:50.911 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.911 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.911 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.912 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s3 49/31 ECSubWrite(tid=25, reqid=osd.10.0:24, at_version=0'0, trim_to=0'0, roll_forward_to=49'15))", 2023-10-20T03:35:50.913 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.408468+0000", 2023-10-20T03:35:50.913 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 19.287017480999999, 2023-10-20T03:35:50.913 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014029283, 2023-10-20T03:35:50.914 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.914 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.914 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.914 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.915 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.915 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408468+0000", 2023-10-20T03:35:50.915 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.916 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.916 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.916 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408468+0000", 2023-10-20T03:35:50.916 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.917 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.917 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.917 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408470+0000", 2023-10-20T03:35:50.918 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1680000000000001e-06 2023-10-20T03:35:50.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.919 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.919 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408486+0000", 2023-10-20T03:35:50.919 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6209999999999999e-05 2023-10-20T03:35:50.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.920 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.920 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408502+0000", 2023-10-20T03:35:50.920 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6351e-05 2023-10-20T03:35:50.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.921 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.921 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.921 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408514+0000", 2023-10-20T03:35:50.921 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1364e-05 2023-10-20T03:35:50.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.922 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.923 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408748+0000", 2023-10-20T03:35:50.923 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000234088 2023-10-20T03:35:50.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.923 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.924 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.924 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408763+0000", 2023-10-20T03:35:50.924 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5586999999999998e-05 2023-10-20T03:35:50.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.925 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.925 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.925 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.422444+0000", 2023-10-20T03:35:50.926 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013680701999999999 2023-10-20T03:35:50.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.926 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.927 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.422498+0000", 2023-10-20T03:35:50.927 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3813000000000001e-05 2023-10-20T03:35:50.927 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.927 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.928 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.929 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.fs0 53/47 ECSubWrite(tid=57, reqid=client.4570.0:408, at_version=53'31, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:50.929 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:35.448532+0000", 2023-10-20T03:35:50.929 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.246954071999999, 2023-10-20T03:35:50.929 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012712822, 2023-10-20T03:35:50.930 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.930 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.930 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.931 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.931 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.931 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.448532+0000", 2023-10-20T03:35:50.931 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.932 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.932 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.933 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.448532+0000", 2023-10-20T03:35:50.933 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.933 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.934 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.934 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.934 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.448532+0000", 2023-10-20T03:35:50.935 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2799999999999996e-07 2023-10-20T03:35:50.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.935 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.935 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.936 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449112+0000", 2023-10-20T03:35:50.936 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00057979400000000001 2023-10-20T03:35:50.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.937 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.937 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.937 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449124+0000", 2023-10-20T03:35:50.937 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1613999999999999e-05 2023-10-20T03:35:50.938 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.938 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.938 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.939 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449129+0000", 2023-10-20T03:35:50.939 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6049999999999997e-06 2023-10-20T03:35:50.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.940 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.940 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449245+0000", 2023-10-20T03:35:50.940 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011568199999999999 2023-10-20T03:35:50.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.941 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.941 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.941 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449261+0000", 2023-10-20T03:35:50.941 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6376999999999999e-05 2023-10-20T03:35:50.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.942 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.942 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.942 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.461172+0000", 2023-10-20T03:35:50.943 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01191096 2023-10-20T03:35:50.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.943 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.944 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.944 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.461245+0000", 2023-10-20T03:35:50.944 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2261999999999994e-05 2023-10-20T03:35:50.944 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.945 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.945 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.945 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.946 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.fs0 53/47 ECSubWrite(tid=58, reqid=client.4570.0:409, at_version=53'32, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:50.946 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:35.472456+0000", 2023-10-20T03:35:50.946 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.223029537, 2023-10-20T03:35:50.946 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011134987000000001, 2023-10-20T03:35:50.947 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.947 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.947 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.947 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.947 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.948 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472456+0000", 2023-10-20T03:35:50.948 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.948 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.948 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.949 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.949 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472456+0000", 2023-10-20T03:35:50.949 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.949 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.949 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.950 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.950 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472457+0000", 2023-10-20T03:35:50.950 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7999999999999996e-07 2023-10-20T03:35:50.950 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.950 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.951 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.951 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472728+0000", 2023-10-20T03:35:50.951 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00027096800000000001 2023-10-20T03:35:50.951 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.952 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.952 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.952 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472738+0000", 2023-10-20T03:35:50.952 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0495e-05 2023-10-20T03:35:50.953 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.953 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.953 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.953 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472744+0000", 2023-10-20T03:35:50.953 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0410000000000003e-06 2023-10-20T03:35:50.954 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.954 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.954 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.955 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472867+0000", 2023-10-20T03:35:50.955 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012291400000000001 2023-10-20T03:35:50.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.955 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.956 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.956 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472883+0000", 2023-10-20T03:35:50.956 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5628e-05 2023-10-20T03:35:50.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.956 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.956 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.957 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.483517+0000", 2023-10-20T03:35:50.957 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010634423 2023-10-20T03:35:50.957 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.957 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.958 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.958 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.483591+0000", 2023-10-20T03:35:50.958 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4037999999999998e-05 2023-10-20T03:35:50.958 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.958 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.959 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.959 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.959 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.959 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 60/56 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=60'45, trim_to=0'0, roll_forward_to=57'43))", 2023-10-20T03:35:50.959 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.831061+0000", 2023-10-20T03:35:50.960 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8644254349999994, 2023-10-20T03:35:50.960 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.008194425, 2023-10-20T03:35:50.960 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.960 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.960 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.961 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.961 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.961 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831061+0000", 2023-10-20T03:35:50.961 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.962 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.962 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.962 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831061+0000", 2023-10-20T03:35:50.962 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.963 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.963 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.963 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.963 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831061+0000", 2023-10-20T03:35:50.963 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3799999999999998e-07 2023-10-20T03:35:50.964 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.964 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.964 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.965 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831106+0000", 2023-10-20T03:35:50.965 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4922000000000001e-05 2023-10-20T03:35:50.965 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.965 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.965 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.966 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831127+0000", 2023-10-20T03:35:50.966 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1013999999999999e-05 2023-10-20T03:35:50.966 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.967 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.967 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.967 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831144+0000", 2023-10-20T03:35:50.967 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6935e-05 2023-10-20T03:35:50.968 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.968 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.968 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.968 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831295+0000", 2023-10-20T03:35:50.969 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015154599999999999 2023-10-20T03:35:50.969 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.969 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.969 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.970 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831319+0000", 2023-10-20T03:35:50.970 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3764000000000002e-05 2023-10-20T03:35:50.970 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.970 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.971 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.971 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.839123+0000", 2023-10-20T03:35:50.971 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.007803918 2023-10-20T03:35:50.971 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.972 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.972 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.972 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.839255+0000", 2023-10-20T03:35:50.972 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013198800000000001 2023-10-20T03:35:50.973 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.973 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.973 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.973 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.974 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.974 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.fs0 60/47 ECSubWrite(tid=87, reqid=osd.3.0:87, at_version=60'54, trim_to=0'0, roll_forward_to=60'50))", 2023-10-20T03:35:50.974 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.885693+0000", 2023-10-20T03:35:50.975 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.8097928890000006, 2023-10-20T03:35:50.975 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0066511110000000003, 2023-10-20T03:35:50.975 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.975 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.976 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.976 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.976 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.976 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.885693+0000", 2023-10-20T03:35:50.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.885693+0000", 2023-10-20T03:35:50.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.979 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.980 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.885695+0000", 2023-10-20T03:35:50.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7430000000000001e-06 2023-10-20T03:35:50.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.980 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.980 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.981 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.885736+0000", 2023-10-20T03:35:50.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1606000000000001e-05 2023-10-20T03:35:50.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.981 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.981 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.982 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.885767+0000", 2023-10-20T03:35:50.982 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0083e-05 2023-10-20T03:35:50.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.982 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.983 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.885864+0000", 2023-10-20T03:35:50.983 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7201000000000007e-05 2023-10-20T03:35:50.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.984 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.984 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.885958+0000", 2023-10-20T03:35:50.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4413999999999994e-05 2023-10-20T03:35:50.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.885967+0000", 2023-10-20T03:35:50.985 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7730000000000005e-06 2023-10-20T03:35:50.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.986 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.986 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:50.986 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.892311+0000", 2023-10-20T03:35:50.987 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0063445979999999999 2023-10-20T03:35:50.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.987 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.987 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:50.987 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.892344+0000", 2023-10-20T03:35:50.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2693e-05 2023-10-20T03:35:50.988 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.988 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:50.988 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:50.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.989 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.7s6 60/31 ECSubWrite(tid=200, reqid=client.4570.0:459, at_version=60'9, trim_to=0'0, roll_forward_to=38'7))", 2023-10-20T03:35:50.989 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.897194+0000", 2023-10-20T03:35:50.990 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.7982922220000006, 2023-10-20T03:35:50.990 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083457080000000003, 2023-10-20T03:35:50.990 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:50.990 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:50.990 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:50.991 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.991 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:50.991 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897194+0000", 2023-10-20T03:35:50.991 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.991 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.992 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.992 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:50.992 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897194+0000", 2023-10-20T03:35:50.992 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:50.993 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.993 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.993 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:50.993 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897194+0000", 2023-10-20T03:35:50.993 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4399999999999999e-07 2023-10-20T03:35:50.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.994 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.994 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:50.994 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897530+0000", 2023-10-20T03:35:50.994 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00033525200000000002 2023-10-20T03:35:50.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.995 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.995 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:50.995 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897545+0000", 2023-10-20T03:35:50.996 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5299999999999999e-05 2023-10-20T03:35:50.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.996 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.997 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:50.997 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897553+0000", 2023-10-20T03:35:50.997 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7940000000000003e-06 2023-10-20T03:35:50.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.997 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.998 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:50.998 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897629+0000", 2023-10-20T03:35:50.998 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5766000000000006e-05 2023-10-20T03:35:50.998 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:50.998 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:50.999 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:50.999 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897644+0000", 2023-10-20T03:35:50.999 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5834000000000001e-05 2023-10-20T03:35:50.999 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.000 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.000 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.000 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.905485+0000", 2023-10-20T03:35:51.000 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0078405750000000007 2023-10-20T03:35:51.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.001 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.001 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.001 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.905539+0000", 2023-10-20T03:35:51.001 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4443e-05 2023-10-20T03:35:51.002 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.002 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.002 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.003 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.003 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.bs3 60/56 ECSubWrite(tid=204, reqid=client.4570.0:470, at_version=60'24, trim_to=0'0, roll_forward_to=60'22))", 2023-10-20T03:35:51.003 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.923867+0000", 2023-10-20T03:35:51.004 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 8.7716185620000005, 2023-10-20T03:35:51.004 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014718472999999999, 2023-10-20T03:35:51.004 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.004 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.005 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.005 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.005 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.005 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.923867+0000", 2023-10-20T03:35:51.005 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.006 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.006 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.006 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.923867+0000", 2023-10-20T03:35:51.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.007 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.923868+0000", 2023-10-20T03:35:51.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.063e-06 2023-10-20T03:35:51.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.009 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.009 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924029+0000", 2023-10-20T03:35:51.009 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000160927 2023-10-20T03:35:51.009 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.010 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.010 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.010 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924045+0000", 2023-10-20T03:35:51.010 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5874000000000001e-05 2023-10-20T03:35:51.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.011 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.011 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.012 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924054+0000", 2023-10-20T03:35:51.012 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7320000000000004e-06 2023-10-20T03:35:51.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.013 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.013 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924173+0000", 2023-10-20T03:35:51.013 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000118713 2023-10-20T03:35:51.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.014 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.014 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.015 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924190+0000", 2023-10-20T03:35:51.015 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7280000000000001e-05 2023-10-20T03:35:51.015 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.016 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.016 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.017 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.938488+0000", 2023-10-20T03:35:51.017 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014298132 2023-10-20T03:35:51.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.018 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.018 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.018 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.938586+0000", 2023-10-20T03:35:51.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7751999999999995e-05 2023-10-20T03:35:51.019 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.019 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.019 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.020 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.020 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.020 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:51.022 INFO:tasks.rados.rados.0.smithi170.stdout:229: done (0 left) 2023-10-20T03:35:51.023 INFO:tasks.rados.rados.0.smithi170.stdout:230: copy_from oid 2 from oid 38 current snap is 18 2023-10-20T03:35:51.023 INFO:tasks.rados.rados.0.smithi170.stdout:231: delete oid 13 current snap is 18 2023-10-20T03:35:51.023 INFO:tasks.rados.rados.0.smithi170.stdout:231: done (1 left) 2023-10-20T03:35:51.023 INFO:tasks.rados.rados.0.smithi170.stdout:232: snap_remove snap 16 2023-10-20T03:35:51.024 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-10-20T03:35:51.025 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2023-10-20T03:35:51.026 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:51.021+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:51.123 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:51.120+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:51.200 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:51.201 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:51.201 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:51.201 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:51.215 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-10-20T03:35:51.224 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:51.222+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:51.325 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:51.325+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:51.334 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:51.334 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:51.334 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:35:51.334 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:35:51.335 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:51.348 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-10-20T03:35:51.438 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:51.437+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:51.521 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:51.521 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:51.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:51.522 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:35:51.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.522 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s3 36/31 ECSubWrite(tid=21, reqid=client.4570.0:198, at_version=36'13, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:51.523 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.265679+0000", 2023-10-20T03:35:51.523 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 33.252697241, 2023-10-20T03:35:51.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0082907580000000005, 2023-10-20T03:35:51.523 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.523 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.524 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.524 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265679+0000", 2023-10-20T03:35:51.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.525 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.525 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.525 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265679+0000", 2023-10-20T03:35:51.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.526 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.265680+0000", 2023-10-20T03:35:51.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.04e-07 2023-10-20T03:35:51.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266116+0000", 2023-10-20T03:35:51.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00043625499999999998 2023-10-20T03:35:51.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266135+0000", 2023-10-20T03:35:51.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8974000000000002e-05 2023-10-20T03:35:51.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266146+0000", 2023-10-20T03:35:51.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1523e-05 2023-10-20T03:35:51.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266263+0000", 2023-10-20T03:35:51.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000116945 2023-10-20T03:35:51.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266275+0000", 2023-10-20T03:35:51.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1941e-05 2023-10-20T03:35:51.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.273944+0000", 2023-10-20T03:35:51.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0076690689999999997 2023-10-20T03:35:51.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.273970+0000", 2023-10-20T03:35:51.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5846999999999998e-05 2023-10-20T03:35:51.536 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.536 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.536 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.537 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 44/31 ECSubWrite(tid=19, reqid=client.4570.0:324, at_version=44'10, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:35:51.537 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.381110+0000", 2023-10-20T03:35:51.537 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.137266752999999, 2023-10-20T03:35:51.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0096630399999999995, 2023-10-20T03:35:51.538 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.538 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.538 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.538 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381110+0000", 2023-10-20T03:35:51.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381110+0000", 2023-10-20T03:35:51.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381111+0000", 2023-10-20T03:35:51.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5300000000000002e-07 2023-10-20T03:35:51.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.542 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381249+0000", 2023-10-20T03:35:51.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000138088 2023-10-20T03:35:51.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381265+0000", 2023-10-20T03:35:51.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5886999999999999e-05 2023-10-20T03:35:51.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381274+0000", 2023-10-20T03:35:51.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0550000000000005e-06 2023-10-20T03:35:51.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381348+0000", 2023-10-20T03:35:51.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4064000000000001e-05 2023-10-20T03:35:51.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381358+0000", 2023-10-20T03:35:51.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0176999999999999e-05 2023-10-20T03:35:51.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.390717+0000", 2023-10-20T03:35:51.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093585939999999996 2023-10-20T03:35:51.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.549 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.390773+0000", 2023-10-20T03:35:51.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6221999999999998e-05 2023-10-20T03:35:51.549 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.549 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.550 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.550 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 44/31 ECSubWrite(tid=20, reqid=client.4570.0:325, at_version=44'11, trim_to=0'0, roll_forward_to=44'9))", 2023-10-20T03:35:51.550 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.398572+0000", 2023-10-20T03:35:51.551 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.119804345999999, 2023-10-20T03:35:51.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093426800000000008, 2023-10-20T03:35:51.551 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.551 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.551 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.398572+0000", 2023-10-20T03:35:51.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.398572+0000", 2023-10-20T03:35:51.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.398573+0000", 2023-10-20T03:35:51.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2979999999999999e-06 2023-10-20T03:35:51.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.398622+0000", 2023-10-20T03:35:51.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8535999999999997e-05 2023-10-20T03:35:51.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.398639+0000", 2023-10-20T03:35:51.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6614999999999999e-05 2023-10-20T03:35:51.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.398651+0000", 2023-10-20T03:35:51.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2719e-05 2023-10-20T03:35:51.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.398812+0000", 2023-10-20T03:35:51.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000160798 2023-10-20T03:35:51.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.398827+0000", 2023-10-20T03:35:51.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5187e-05 2023-10-20T03:35:51.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.407873+0000", 2023-10-20T03:35:51.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.009045839 2023-10-20T03:35:51.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.407915+0000", 2023-10-20T03:35:51.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1687999999999998e-05 2023-10-20T03:35:51.563 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.563 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.563 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.564 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:51.564 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.401231+0000", 2023-10-20T03:35:51.564 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.117145547, 2023-10-20T03:35:51.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010804612, 2023-10-20T03:35:51.565 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.565 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.565 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401231+0000", 2023-10-20T03:35:51.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401231+0000", 2023-10-20T03:35:51.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401232+0000", 2023-10-20T03:35:51.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.052e-06 2023-10-20T03:35:51.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401438+0000", 2023-10-20T03:35:51.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020594299999999999 2023-10-20T03:35:51.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401458+0000", 2023-10-20T03:35:51.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0398e-05 2023-10-20T03:35:51.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401511+0000", 2023-10-20T03:35:51.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3022000000000002e-05 2023-10-20T03:35:51.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.572 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401606+0000", 2023-10-20T03:35:51.572 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4421999999999998e-05 2023-10-20T03:35:51.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401619+0000", 2023-10-20T03:35:51.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3410999999999999e-05 2023-10-20T03:35:51.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.574 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.574 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.411992+0000", 2023-10-20T03:35:51.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010372354 2023-10-20T03:35:51.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.412036+0000", 2023-10-20T03:35:51.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.401e-05 2023-10-20T03:35:51.576 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.576 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.576 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.577 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 44/31 ECSubWrite(tid=53, reqid=client.4570.0:335, at_version=44'20, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:51.577 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.402821+0000", 2023-10-20T03:35:51.577 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.115555484000001, 2023-10-20T03:35:51.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099566840000000004, 2023-10-20T03:35:51.578 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.578 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.578 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.579 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.579 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.402821+0000", 2023-10-20T03:35:51.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.580 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.580 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.402821+0000", 2023-10-20T03:35:51.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.402822+0000", 2023-10-20T03:35:51.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8400000000000003e-07 2023-10-20T03:35:51.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.583 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.402852+0000", 2023-10-20T03:35:51.583 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0224999999999999e-05 2023-10-20T03:35:51.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.584 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.584 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.585 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.402868+0000", 2023-10-20T03:35:51.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5824999999999998e-05 2023-10-20T03:35:51.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.586 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.586 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.587 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.402878+0000", 2023-10-20T03:35:51.587 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0057e-05 2023-10-20T03:35:51.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.410800+0000", 2023-10-20T03:35:51.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079223220000000007 2023-10-20T03:35:51.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.589 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.589 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.410811+0000", 2023-10-20T03:35:51.590 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0529e-05 2023-10-20T03:35:51.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.412707+0000", 2023-10-20T03:35:51.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0018963859999999999 2023-10-20T03:35:51.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.591 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.591 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.412778+0000", 2023-10-20T03:35:51.592 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0555999999999993e-05 2023-10-20T03:35:51.592 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.592 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.592 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.592 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.593 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 44/31 ECSubWrite(tid=54, reqid=client.4570.0:336, at_version=44'21, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:51.593 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.405081+0000", 2023-10-20T03:35:51.594 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.113295274999999, 2023-10-20T03:35:51.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083306530000000004, 2023-10-20T03:35:51.594 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.595 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.595 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.595 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.596 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405081+0000", 2023-10-20T03:35:51.596 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.596 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405081+0000", 2023-10-20T03:35:51.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.598 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.598 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.598 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405082+0000", 2023-10-20T03:35:51.598 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7300000000000004e-07 2023-10-20T03:35:51.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.599 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.599 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.599 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405108+0000", 2023-10-20T03:35:51.599 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5371000000000002e-05 2023-10-20T03:35:51.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.600 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.600 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405125+0000", 2023-10-20T03:35:51.600 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7130999999999999e-05 2023-10-20T03:35:51.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.601 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.601 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.601 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405137+0000", 2023-10-20T03:35:51.602 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2221e-05 2023-10-20T03:35:51.602 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.602 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.602 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.602 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.411715+0000", 2023-10-20T03:35:51.603 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0065784930000000004 2023-10-20T03:35:51.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.603 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.603 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.411728+0000", 2023-10-20T03:35:51.604 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2451e-05 2023-10-20T03:35:51.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.604 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.604 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.604 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413359+0000", 2023-10-20T03:35:51.605 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001630804 2023-10-20T03:35:51.605 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.605 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.605 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.606 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413412+0000", 2023-10-20T03:35:51.606 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3208999999999998e-05 2023-10-20T03:35:51.606 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.606 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.606 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.607 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:35:51.607 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.392154+0000", 2023-10-20T03:35:51.607 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.126222534, 2023-10-20T03:35:51.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055444475, 2023-10-20T03:35:51.608 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.608 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.608 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392154+0000", 2023-10-20T03:35:51.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392154+0000", 2023-10-20T03:35:51.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.611 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.611 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392156+0000", 2023-10-20T03:35:51.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0499999999999999e-06 2023-10-20T03:35:51.612 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.612 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.612 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.612 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392978+0000", 2023-10-20T03:35:51.613 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00082210300000000003 2023-10-20T03:35:51.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393026+0000", 2023-10-20T03:35:51.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8306000000000001e-05 2023-10-20T03:35:51.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393053+0000", 2023-10-20T03:35:51.617 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6872e-05 2023-10-20T03:35:51.617 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.617 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.617 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.618 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393248+0000", 2023-10-20T03:35:51.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019507500000000001 2023-10-20T03:35:51.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393331+0000", 2023-10-20T03:35:51.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2701999999999995e-05 2023-10-20T03:35:51.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.620 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.620 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.447508+0000", 2023-10-20T03:35:51.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054176766000000001 2023-10-20T03:35:51.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.621 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.621 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.621 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.447598+0000", 2023-10-20T03:35:51.622 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0600999999999995e-05 2023-10-20T03:35:51.622 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.622 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.622 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.625 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 49/31 ECSubWrite(tid=39, reqid=client.4570.0:390, at_version=49'23, trim_to=0'0, roll_forward_to=47'21))", 2023-10-20T03:35:51.626 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.413323+0000", 2023-10-20T03:35:51.626 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.105053412, 2023-10-20T03:35:51.626 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012748729, 2023-10-20T03:35:51.626 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.626 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.627 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.627 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.627 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.413323+0000", 2023-10-20T03:35:51.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.413323+0000", 2023-10-20T03:35:51.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.413324+0000", 2023-10-20T03:35:51.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0699999999999999e-06 2023-10-20T03:35:51.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.630 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.631 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414070+0000", 2023-10-20T03:35:51.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00074601599999999995 2023-10-20T03:35:51.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.632 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414089+0000", 2023-10-20T03:35:51.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9199999999999999e-05 2023-10-20T03:35:51.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414102+0000", 2023-10-20T03:35:51.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2099000000000001e-05 2023-10-20T03:35:51.633 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.633 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.634 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414201+0000", 2023-10-20T03:35:51.634 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9931000000000003e-05 2023-10-20T03:35:51.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.634 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.635 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414214+0000", 2023-10-20T03:35:51.635 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2652e-05 2023-10-20T03:35:51.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.635 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.636 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.636 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.426026+0000", 2023-10-20T03:35:51.636 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011811845 2023-10-20T03:35:51.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.637 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.426072+0000", 2023-10-20T03:35:51.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5915999999999999e-05 2023-10-20T03:35:51.638 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.638 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.638 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.639 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.es3 54/31 ECSubWrite(tid=37, reqid=client.4570.0:414, at_version=54'20, trim_to=0'0, roll_forward_to=39'19))", 2023-10-20T03:35:51.639 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:36.433174+0000", 2023-10-20T03:35:51.639 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.085202985, 2023-10-20T03:35:51.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010871729, 2023-10-20T03:35:51.640 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.640 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.640 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.433174+0000", 2023-10-20T03:35:51.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.642 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.433174+0000", 2023-10-20T03:35:51.642 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.643 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.433175+0000", 2023-10-20T03:35:51.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8300000000000001e-06 2023-10-20T03:35:51.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.644 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.644 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.433343+0000", 2023-10-20T03:35:51.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016747999999999999 2023-10-20T03:35:51.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.646 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.433390+0000", 2023-10-20T03:35:51.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6859e-05 2023-10-20T03:35:51.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.647 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.647 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.433415+0000", 2023-10-20T03:35:51.647 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5216e-05 2023-10-20T03:35:51.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.647 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.648 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.648 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.433699+0000", 2023-10-20T03:35:51.648 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028422099999999997 2023-10-20T03:35:51.648 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.648 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.433727+0000", 2023-10-20T03:35:51.649 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7483000000000001e-05 2023-10-20T03:35:51.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.443937+0000", 2023-10-20T03:35:51.650 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010210581999999999 2023-10-20T03:35:51.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.651 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.651 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.651 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.444045+0000", 2023-10-20T03:35:51.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000108058 2023-10-20T03:35:51.652 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.652 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.652 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.653 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 54/31 ECSubWrite(tid=65, reqid=client.4570.0:418, at_version=54'32, trim_to=0'0, roll_forward_to=51'31))", 2023-10-20T03:35:51.653 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:36.458762+0000", 2023-10-20T03:35:51.653 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.059614655000001, 2023-10-20T03:35:51.653 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015096593, 2023-10-20T03:35:51.654 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.654 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.654 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.654 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.654 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.458762+0000", 2023-10-20T03:35:51.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.655 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.655 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.458762+0000", 2023-10-20T03:35:51.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.458764+0000", 2023-10-20T03:35:51.657 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6700000000000001e-06 2023-10-20T03:35:51.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.657 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.657 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.459218+0000", 2023-10-20T03:35:51.658 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00045411099999999999 2023-10-20T03:35:51.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.658 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.659 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.459248+0000", 2023-10-20T03:35:51.659 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0828999999999999e-05 2023-10-20T03:35:51.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.659 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.659 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.660 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.459264+0000", 2023-10-20T03:35:51.660 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5143000000000001e-05 2023-10-20T03:35:51.660 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.660 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.660 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.661 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.459379+0000", 2023-10-20T03:35:51.661 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011584800000000001 2023-10-20T03:35:51.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.661 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.662 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.459398+0000", 2023-10-20T03:35:51.662 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8185999999999998e-05 2023-10-20T03:35:51.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.663 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.663 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.473773+0000", 2023-10-20T03:35:51.663 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014375221000000001 2023-10-20T03:35:51.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.664 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.664 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.664 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.473858+0000", 2023-10-20T03:35:51.664 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5585000000000001e-05 2023-10-20T03:35:51.664 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.665 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.665 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.665 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.666 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.666 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 54/31 ECSubWrite(tid=66, reqid=client.4570.0:419, at_version=54'33, trim_to=0'0, roll_forward_to=51'31))", 2023-10-20T03:35:51.666 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:36.475739+0000", 2023-10-20T03:35:51.667 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.042637484, 2023-10-20T03:35:51.667 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010418985, 2023-10-20T03:35:51.668 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.668 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.668 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.669 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.670 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.475739+0000", 2023-10-20T03:35:51.670 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.671 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.671 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.672 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.475739+0000", 2023-10-20T03:35:51.672 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.672 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.673 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.673 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.475741+0000", 2023-10-20T03:35:51.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5069999999999999e-06 2023-10-20T03:35:51.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.676 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.475941+0000", 2023-10-20T03:35:51.676 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000200655 2023-10-20T03:35:51.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.677 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.677 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.475972+0000", 2023-10-20T03:35:51.678 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1244000000000001e-05 2023-10-20T03:35:51.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.679 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.679 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.475987+0000", 2023-10-20T03:35:51.680 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4902000000000001e-05 2023-10-20T03:35:51.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.680 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.681 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.476131+0000", 2023-10-20T03:35:51.681 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000143824 2023-10-20T03:35:51.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.682 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.682 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.476160+0000", 2023-10-20T03:35:51.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8586e-05 2023-10-20T03:35:51.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.683 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.683 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.486066+0000", 2023-10-20T03:35:51.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099061619999999996 2023-10-20T03:35:51.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.486158+0000", 2023-10-20T03:35:51.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2105e-05 2023-10-20T03:35:51.685 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.685 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.685 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.686 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 54/31 ECSubWrite(tid=67, reqid=client.4570.0:420, at_version=54'34, trim_to=0'0, roll_forward_to=54'32))", 2023-10-20T03:35:51.686 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:36.487800+0000", 2023-10-20T03:35:51.686 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.030576777, 2023-10-20T03:35:51.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.007683311, 2023-10-20T03:35:51.687 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.687 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.687 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.688 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.487800+0000", 2023-10-20T03:35:51.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.689 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.487800+0000", 2023-10-20T03:35:51.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.690 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.691 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.487801+0000", 2023-10-20T03:35:51.692 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4699999999999999e-06 2023-10-20T03:35:51.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.693 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.693 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.487948+0000", 2023-10-20T03:35:51.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014666000000000001 2023-10-20T03:35:51.694 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.694 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.487979+0000", 2023-10-20T03:35:51.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1130000000000002e-05 2023-10-20T03:35:51.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.487995+0000", 2023-10-20T03:35:51.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5702999999999999e-05 2023-10-20T03:35:51.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.488138+0000", 2023-10-20T03:35:51.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014322400000000001 2023-10-20T03:35:51.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.488166+0000", 2023-10-20T03:35:51.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7994999999999999e-05 2023-10-20T03:35:51.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.699 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.699 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.495386+0000", 2023-10-20T03:35:51.699 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0072196179999999997 2023-10-20T03:35:51.699 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.700 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.700 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.495483+0000", 2023-10-20T03:35:51.700 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7510999999999995e-05 2023-10-20T03:35:51.700 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.701 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.701 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.701 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.1s1 60/31 ECSubWrite(tid=190, reqid=client.4570.0:452, at_version=60'54, trim_to=0'0, roll_forward_to=58'48))", 2023-10-20T03:35:51.702 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.855900+0000", 2023-10-20T03:35:51.702 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.6624766740000005, 2023-10-20T03:35:51.702 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010972529999999999, 2023-10-20T03:35:51.702 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.703 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.703 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.703 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.703 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.704 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.855900+0000", 2023-10-20T03:35:51.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.704 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.705 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.855900+0000", 2023-10-20T03:35:51.705 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.855901+0000", 2023-10-20T03:35:51.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0950000000000001e-06 2023-10-20T03:35:51.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.707 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.707 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.856180+0000", 2023-10-20T03:35:51.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000279175 2023-10-20T03:35:51.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.856197+0000", 2023-10-20T03:35:51.709 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6509999999999999e-05 2023-10-20T03:35:51.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.709 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.710 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.856205+0000", 2023-10-20T03:35:51.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7579999999999998e-06 2023-10-20T03:35:51.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.711 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.711 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.856322+0000", 2023-10-20T03:35:51.711 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011708400000000001 2023-10-20T03:35:51.712 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.712 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.712 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.712 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.856341+0000", 2023-10-20T03:35:51.712 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8808999999999999e-05 2023-10-20T03:35:51.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.713 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.866836+0000", 2023-10-20T03:35:51.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010494684000000001 2023-10-20T03:35:51.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.866872+0000", 2023-10-20T03:35:51.715 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6415000000000003e-05 2023-10-20T03:35:51.715 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.715 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.715 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.716 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.716 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.1s1 60/31 ECSubWrite(tid=195, reqid=client.4570.0:453, at_version=60'55, trim_to=0'0, roll_forward_to=60'52))", 2023-10-20T03:35:51.716 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.877055+0000", 2023-10-20T03:35:51.716 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.6413220020000008, 2023-10-20T03:35:51.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092683279999999993, 2023-10-20T03:35:51.717 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.717 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.718 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.719 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.877055+0000", 2023-10-20T03:35:51.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.877055+0000", 2023-10-20T03:35:51.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.722 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.722 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.877056+0000", 2023-10-20T03:35:51.722 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7199999999999997e-07 2023-10-20T03:35:51.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.722 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.723 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.723 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.877351+0000", 2023-10-20T03:35:51.723 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000295953 2023-10-20T03:35:51.723 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.723 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.724 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.724 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.877374+0000", 2023-10-20T03:35:51.724 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2140000000000001e-05 2023-10-20T03:35:51.724 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.725 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.725 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.725 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.877387+0000", 2023-10-20T03:35:51.725 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3414000000000001e-05 2023-10-20T03:35:51.725 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.726 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.726 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.726 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.877523+0000", 2023-10-20T03:35:51.726 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013638100000000001 2023-10-20T03:35:51.726 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.727 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.727 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.727 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.877539+0000", 2023-10-20T03:35:51.727 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.517e-05 2023-10-20T03:35:51.728 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.728 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.728 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.728 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.886241+0000", 2023-10-20T03:35:51.728 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0087023570000000008 2023-10-20T03:35:51.729 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.729 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.729 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.729 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.886323+0000", 2023-10-20T03:35:51.730 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1940999999999994e-05 2023-10-20T03:35:51.730 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.730 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.730 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.731 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.731 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.731 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.1s1 60/31 ECSubWrite(tid=199, reqid=client.4570.0:454, at_version=60'56, trim_to=0'0, roll_forward_to=60'52))", 2023-10-20T03:35:51.731 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.894519+0000", 2023-10-20T03:35:51.732 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.6238579489999996, 2023-10-20T03:35:51.732 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0078259019999999992, 2023-10-20T03:35:51.732 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.732 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.732 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.733 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.733 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.733 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.894519+0000", 2023-10-20T03:35:51.733 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.734 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.734 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.734 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.734 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.894519+0000", 2023-10-20T03:35:51.734 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.735 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.735 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.735 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.735 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.894520+0000", 2023-10-20T03:35:51.736 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0699999999999999e-06 2023-10-20T03:35:51.736 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.736 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.737 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.737 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.894598+0000", 2023-10-20T03:35:51.737 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8038000000000001e-05 2023-10-20T03:35:51.737 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.737 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.738 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.738 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.894615+0000", 2023-10-20T03:35:51.738 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6830999999999999e-05 2023-10-20T03:35:51.738 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.739 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.739 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.739 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.894623+0000", 2023-10-20T03:35:51.739 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.8370000000000003e-06 2023-10-20T03:35:51.740 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.740 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.740 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.740 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.894764+0000", 2023-10-20T03:35:51.740 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014027500000000001 2023-10-20T03:35:51.741 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.741 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.741 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.741 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.894783+0000", 2023-10-20T03:35:51.742 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9732999999999999e-05 2023-10-20T03:35:51.742 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.742 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.742 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.742 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.902262+0000", 2023-10-20T03:35:51.743 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074786269999999998 2023-10-20T03:35:51.743 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.743 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.743 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.744 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.902344+0000", 2023-10-20T03:35:51.744 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2490999999999993e-05 2023-10-20T03:35:51.744 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.744 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.745 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.745 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.745 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.745 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.bs5 60/56 ECSubWrite(tid=208, reqid=client.4570.0:471, at_version=60'25, trim_to=0'0, roll_forward_to=60'22))", 2023-10-20T03:35:51.745 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.933491+0000", 2023-10-20T03:35:51.746 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.5848856659999999, 2023-10-20T03:35:51.746 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015929024999999999, 2023-10-20T03:35:51.746 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.747 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.747 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.747 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.747 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.748 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933491+0000", 2023-10-20T03:35:51.748 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.748 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.748 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.749 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.749 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933491+0000", 2023-10-20T03:35:51.749 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.749 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.749 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.750 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.750 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933492+0000", 2023-10-20T03:35:51.750 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.331e-06 2023-10-20T03:35:51.750 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.751 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933641+0000", 2023-10-20T03:35:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014902599999999999 2023-10-20T03:35:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.752 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933669+0000", 2023-10-20T03:35:51.753 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8107999999999999e-05 2023-10-20T03:35:51.753 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.753 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.754 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.754 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933685+0000", 2023-10-20T03:35:51.754 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5319999999999999e-05 2023-10-20T03:35:51.755 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.755 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.756 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.756 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933835+0000", 2023-10-20T03:35:51.756 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015027700000000001 2023-10-20T03:35:51.757 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.757 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.757 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.757 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933852+0000", 2023-10-20T03:35:51.758 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6965000000000001e-05 2023-10-20T03:35:51.758 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.759 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.759 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.759 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.949309+0000", 2023-10-20T03:35:51.759 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015457256000000001 2023-10-20T03:35:51.760 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.760 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.760 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.760 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.949420+0000", 2023-10-20T03:35:51.760 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000110742 2023-10-20T03:35:51.761 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.761 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.761 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.761 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.761 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.762 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.bs5 60/56 ECSubWrite(tid=211, reqid=client.4570.0:472, at_version=60'26, trim_to=0'0, roll_forward_to=60'22))", 2023-10-20T03:35:51.762 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.939803+0000", 2023-10-20T03:35:51.762 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.5785739499999991, 2023-10-20T03:35:51.762 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013356855000000001, 2023-10-20T03:35:51.763 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.763 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.763 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.763 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.763 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.764 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.939803+0000", 2023-10-20T03:35:51.764 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.764 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.764 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.764 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.765 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.939803+0000", 2023-10-20T03:35:51.765 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.765 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.766 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.766 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.766 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.939805+0000", 2023-10-20T03:35:51.766 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1569999999999998e-06 2023-10-20T03:35:51.766 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.767 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.767 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.767 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.940088+0000", 2023-10-20T03:35:51.767 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028281699999999999 2023-10-20T03:35:51.768 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.768 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.768 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.768 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.940123+0000", 2023-10-20T03:35:51.768 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5930999999999999e-05 2023-10-20T03:35:51.769 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.769 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.769 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.769 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.940145+0000", 2023-10-20T03:35:51.770 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1299999999999999e-05 2023-10-20T03:35:51.770 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.770 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.770 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.771 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.941352+0000", 2023-10-20T03:35:51.771 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001207381 2023-10-20T03:35:51.771 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.771 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.771 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.772 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.941370+0000", 2023-10-20T03:35:51.772 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7666999999999999e-05 2023-10-20T03:35:51.772 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.772 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.773 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.773 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.953053+0000", 2023-10-20T03:35:51.773 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011683321 2023-10-20T03:35:51.773 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.773 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.774 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.774 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.953159+0000", 2023-10-20T03:35:51.774 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010628099999999999 2023-10-20T03:35:51.774 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.775 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.775 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.775 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.775 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.775 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.bs5 60/56 ECSubWrite(tid=212, reqid=client.4570.0:473, at_version=60'27, trim_to=0'0, roll_forward_to=60'22))", 2023-10-20T03:35:51.776 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.945490+0000", 2023-10-20T03:35:51.776 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 9.5728867009999998, 2023-10-20T03:35:51.776 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010784748, 2023-10-20T03:35:51.776 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.777 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.777 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.777 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.777 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.777 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.945490+0000", 2023-10-20T03:35:51.778 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.778 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.778 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.779 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.779 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.945490+0000", 2023-10-20T03:35:51.779 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.779 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.779 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.780 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.780 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.945491+0000", 2023-10-20T03:35:51.780 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.528e-06 2023-10-20T03:35:51.780 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.780 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.781 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.781 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.945632+0000", 2023-10-20T03:35:51.781 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014076299999999999 2023-10-20T03:35:51.781 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.782 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.782 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.782 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.945662+0000", 2023-10-20T03:35:51.782 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9425e-05 2023-10-20T03:35:51.782 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.783 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.783 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.783 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.945676+0000", 2023-10-20T03:35:51.783 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4134000000000001e-05 2023-10-20T03:35:51.784 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.784 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.784 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.784 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.949543+0000", 2023-10-20T03:35:51.785 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0038673819999999999 2023-10-20T03:35:51.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.785 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.785 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.949561+0000", 2023-10-20T03:35:51.786 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7941999999999999e-05 2023-10-20T03:35:51.786 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.786 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.786 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.786 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.956169+0000", 2023-10-20T03:35:51.787 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0066081430000000003 2023-10-20T03:35:51.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.787 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.787 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.787 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.956275+0000", 2023-10-20T03:35:51.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000105431 2023-10-20T03:35:51.788 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.788 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.788 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.788 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.789 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.1s1 64/31 ECSubWrite(tid=247, reqid=client.4570.0:540, at_version=64'64, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:35:51.789 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:45.912362+0000", 2023-10-20T03:35:51.789 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.6060147310000001, 2023-10-20T03:35:51.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.007690232, 2023-10-20T03:35:51.789 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.790 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.790 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.790 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.790 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:45.912362+0000", 2023-10-20T03:35:51.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.791 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.791 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:45.912362+0000", 2023-10-20T03:35:51.792 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.792 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.792 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:45.912363+0000", 2023-10-20T03:35:51.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.11e-06 2023-10-20T03:35:51.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.793 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.793 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:45.912537+0000", 2023-10-20T03:35:51.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017457200000000001 2023-10-20T03:35:51.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:45.912552+0000", 2023-10-20T03:35:51.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4214e-05 2023-10-20T03:35:51.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:45.912559+0000", 2023-10-20T03:35:51.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3579999999999997e-06 2023-10-20T03:35:51.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.796 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:45.912624+0000", 2023-10-20T03:35:51.797 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5013999999999998e-05 2023-10-20T03:35:51.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.797 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.797 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:45.912638+0000", 2023-10-20T03:35:51.798 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3682999999999999e-05 2023-10-20T03:35:51.798 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.798 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.798 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.799 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:45.920004+0000", 2023-10-20T03:35:51.799 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073665420000000002 2023-10-20T03:35:51.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.799 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.799 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.800 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:45.920052+0000", 2023-10-20T03:35:51.800 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7738999999999998e-05 2023-10-20T03:35:51.800 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.800 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.800 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.801 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 69/31 ECSubWrite(tid=95, reqid=client.4570.0:552, at_version=69'41, trim_to=0'0, roll_forward_to=56'39))", 2023-10-20T03:35:51.801 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:50.953532+0000", 2023-10-20T03:35:51.802 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.56484413499999997, 2023-10-20T03:35:51.802 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017017713, 2023-10-20T03:35:51.802 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:51.802 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:51.802 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:51.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.803 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:51.803 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.953532+0000", 2023-10-20T03:35:51.803 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.804 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.804 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.804 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:51.804 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.953532+0000", 2023-10-20T03:35:51.804 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:51.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.805 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:51.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.953533+0000", 2023-10-20T03:35:51.806 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1999999999999999e-07 2023-10-20T03:35:51.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.806 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:51.806 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.954582+0000", 2023-10-20T03:35:51.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001049094 2023-10-20T03:35:51.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:51.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.954597+0000", 2023-10-20T03:35:51.808 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5147000000000001e-05 2023-10-20T03:35:51.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.808 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:51.808 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.954606+0000", 2023-10-20T03:35:51.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9570000000000008e-06 2023-10-20T03:35:51.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.809 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.809 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:51.809 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.954681+0000", 2023-10-20T03:35:51.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4901000000000007e-05 2023-10-20T03:35:51.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:51.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.954689+0000", 2023-10-20T03:35:51.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7870000000000007e-06 2023-10-20T03:35:51.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:51.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.970508+0000", 2023-10-20T03:35:51.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015818249999999999 2023-10-20T03:35:51.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:51.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:51.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:51.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.970550+0000", 2023-10-20T03:35:51.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2456999999999997e-05 2023-10-20T03:35:51.813 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.813 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.813 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.814 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:51.814 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:51.814 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:51.816 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-10-20T03:35:51.817 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:51.812+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:51.911 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15] out_osds: [12] dead_osds: [6] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 10, 14, 3, 7, 11, 15] 2023-10-20T03:35:51.911 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-10-20T03:35:51.912 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-10-20T03:35:51.912 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-10-20T03:35:51.916 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:51.914+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:51.939 INFO:tasks.rados.rados.0.smithi170.stdout:232: done (1 left) 2023-10-20T03:35:51.939 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 21 current snap is 18 2023-10-20T03:35:51.939 INFO:tasks.rados.rados.0.smithi170.stdout:233: seq_num 107 ranges {344064=638976,983040=458752,1441792=704512} 2023-10-20T03:35:51.955 INFO:tasks.rados.rados.0.smithi170.stdout:233: writing smithi17039300-21 from 344064 to 983040 tid 1 2023-10-20T03:35:51.962 INFO:tasks.rados.rados.0.smithi170.stdout:233: writing smithi17039300-21 from 983040 to 1441792 tid 2 2023-10-20T03:35:51.975 INFO:tasks.rados.rados.0.smithi170.stdout:233: writing smithi17039300-21 from 1441792 to 2146304 tid 3 2023-10-20T03:35:51.975 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 11 current snap is 18 2023-10-20T03:35:51.975 INFO:tasks.rados.rados.0.smithi170.stdout:234: seq_num 108 ranges {2998272=655360,3653632=606208,4259840=475136,4734976=81920} 2023-10-20T03:35:51.997 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:51.997 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:51.997 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:51.997 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:52.014 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-10-20T03:35:52.018 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:52.016+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:52.018 INFO:tasks.rados.rados.0.smithi170.stdout:234: writing smithi17039300-11 from 2998272 to 3653632 tid 1 2023-10-20T03:35:52.022 INFO:tasks.rados.rados.0.smithi170.stdout:234: writing smithi17039300-11 from 3653632 to 4259840 tid 2 2023-10-20T03:35:52.026 INFO:tasks.rados.rados.0.smithi170.stdout:234: writing smithi17039300-11 from 4259840 to 4734976 tid 3 2023-10-20T03:35:52.027 INFO:tasks.rados.rados.0.smithi170.stdout:234: writing smithi17039300-11 from 4734976 to 4816896 tid 4 2023-10-20T03:35:52.027 INFO:tasks.rados.rados.0.smithi170.stdout:235: rmattr oid 15 current snap is 18 2023-10-20T03:35:52.027 INFO:tasks.rados.rados.0.smithi170.stdout:235: done (3 left) 2023-10-20T03:35:52.028 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 24 current snap is 18 2023-10-20T03:35:52.028 INFO:tasks.rados.rados.0.smithi170.stdout:236: seq_num 109 ranges {393216=671744,1064960=753664,1818624=409600} 2023-10-20T03:35:52.034 INFO:tasks.rados.rados.0.smithi170.stdout:236: writing smithi17039300-24 from 393216 to 1064960 tid 1 2023-10-20T03:35:52.040 INFO:tasks.rados.rados.0.smithi170.stdout:236: writing smithi17039300-24 from 1064960 to 1818624 tid 2 2023-10-20T03:35:52.043 INFO:tasks.rados.rados.0.smithi170.stdout:236: writing smithi17039300-24 from 1818624 to 2228224 tid 3 2023-10-20T03:35:52.043 INFO:tasks.rados.rados.0.smithi170.stdout:237: delete oid 39 current snap is 18 2023-10-20T03:35:52.044 INFO:tasks.rados.rados.0.smithi170.stdout:230: finishing copy_from racing read to smithi17039300-2 2023-10-20T03:35:52.044 INFO:tasks.rados.rados.0.smithi170.stdout:230: finishing copy_from to smithi17039300-2 2023-10-20T03:35:52.044 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 11 (ObjNum 96 snap 13 seq_num 96) dirty exists 2023-10-20T03:35:52.044 INFO:tasks.rados.rados.0.smithi170.stdout:233: finishing write tid 1 to smithi17039300-21 2023-10-20T03:35:52.045 INFO:tasks.rados.rados.0.smithi170.stdout:233: finishing write tid 2 to smithi17039300-21 2023-10-20T03:35:52.045 INFO:tasks.rados.rados.0.smithi170.stdout:233: finishing write tid 3 to smithi17039300-21 2023-10-20T03:35:52.045 INFO:tasks.rados.rados.0.smithi170.stdout:233: finishing write tid 4 to smithi17039300-21 2023-10-20T03:35:52.045 INFO:tasks.rados.rados.0.smithi170.stdout:233: finishing write tid 5 to smithi17039300-21 2023-10-20T03:35:52.045 INFO:tasks.rados.rados.0.smithi170.stdout:233: oid 21 updating version 0 to 41 2023-10-20T03:35:52.046 INFO:tasks.rados.rados.0.smithi170.stdout:233: oid 21 updating version 41 to 43 2023-10-20T03:35:52.046 INFO:tasks.rados.rados.0.smithi170.stdout:233: oid 21 version 43 is already newer than 42 2023-10-20T03:35:52.046 INFO:tasks.rados.rados.0.smithi170.stdout:233: oid 21 version 43 is already newer than 40 2023-10-20T03:35:52.046 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 21 v 43 (ObjNum 107 snap 18 seq_num 107) dirty exists 2023-10-20T03:35:52.047 INFO:tasks.rados.rados.0.smithi170.stdout:233: left oid 21 (ObjNum 107 snap 18 seq_num 107) 2023-10-20T03:35:52.048 INFO:tasks.rados.rados.0.smithi170.stdout:234: finishing write tid 1 to smithi17039300-11 2023-10-20T03:35:52.055 INFO:tasks.rados.rados.0.smithi170.stdout:230: done (4 left) 2023-10-20T03:35:52.056 INFO:tasks.rados.rados.0.smithi170.stdout:233: done (3 left) 2023-10-20T03:35:52.056 INFO:tasks.rados.rados.0.smithi170.stdout:237: done (2 left) 2023-10-20T03:35:52.056 INFO:tasks.rados.rados.0.smithi170.stdout:238: snap_remove snap 17 2023-10-20T03:35:52.118 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:52.114+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:52.144 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:52.144 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:52.144 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:35:52.144 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:35:52.145 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:52.160 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-10-20T03:35:52.219 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:52.216+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:52.238 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:35:52.238 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:52.238 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 70, 2023-10-20T03:35:52.238 INFO:teuthology.orchestra.run.smithi033.stdout: "fsid": "b920f165-e9e7-4c3e-8d7b-cfdc002ae589", 2023-10-20T03:35:52.239 INFO:teuthology.orchestra.run.smithi033.stdout: "created": "2023-10-20T03:34:47.883622+0000", 2023-10-20T03:35:52.239 INFO:teuthology.orchestra.run.smithi033.stdout: "modified": "2023-10-20T03:35:51.927563+0000", 2023-10-20T03:35:52.239 INFO:teuthology.orchestra.run.smithi033.stdout: "last_up_change": "2023-10-20T03:35:11.150103+0000", 2023-10-20T03:35:52.239 INFO:teuthology.orchestra.run.smithi033.stdout: "last_in_change": "2023-10-20T03:35:35.398012+0000", 2023-10-20T03:35:52.240 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-10-20T03:35:52.240 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_num": 5806080, 2023-10-20T03:35:52.240 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_set": [ 2023-10-20T03:35:52.240 INFO:teuthology.orchestra.run.smithi033.stdout: "nodeep-scrub", 2023-10-20T03:35:52.241 INFO:teuthology.orchestra.run.smithi033.stdout: "noscrub", 2023-10-20T03:35:52.241 INFO:teuthology.orchestra.run.smithi033.stdout: "pglog_hardlimit", 2023-10-20T03:35:52.241 INFO:teuthology.orchestra.run.smithi033.stdout: "purged_snapdirs", 2023-10-20T03:35:52.241 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_deletes", 2023-10-20T03:35:52.241 INFO:teuthology.orchestra.run.smithi033.stdout: "sortbitwise" 2023-10-20T03:35:52.242 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-10-20T03:35:52.242 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_version": 6, 2023-10-20T03:35:52.242 INFO:teuthology.orchestra.run.smithi033.stdout: "full_ratio": 0.94999998807907104, 2023-10-20T03:35:52.242 INFO:teuthology.orchestra.run.smithi033.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-10-20T03:35:52.242 INFO:teuthology.orchestra.run.smithi033.stdout: "nearfull_ratio": 0.85000002384185791, 2023-10-20T03:35:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_snapshot": "", 2023-10-20T03:35:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_max": 3, 2023-10-20T03:35:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd": 16, 2023-10-20T03:35:52.243 INFO:teuthology.orchestra.run.smithi033.stdout: "require_min_compat_client": "luminous", 2023-10-20T03:35:52.244 INFO:teuthology.orchestra.run.smithi033.stdout: "min_compat_client": "jewel", 2023-10-20T03:35:52.244 INFO:teuthology.orchestra.run.smithi033.stdout: "require_osd_release": "reef", 2023-10-20T03:35:52.244 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_crimson": false, 2023-10-20T03:35:52.244 INFO:teuthology.orchestra.run.smithi033.stdout: "pools": [ 2023-10-20T03:35:52.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.245 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 1, 2023-10-20T03:35:52.245 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": ".mgr", 2023-10-20T03:35:52.245 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2023-10-20T03:35:00.991869+0000", 2023-10-20T03:35:52.245 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2023-10-20T03:35:52.245 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2023-10-20T03:35:52.246 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2023-10-20T03:35:52.246 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 2, 2023-10-20T03:35:52.246 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 1, 2023-10-20T03:35:52.246 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2023-10-20T03:35:52.247 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2023-10-20T03:35:52.247 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2023-10-20T03:35:52.247 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2023-10-20T03:35:52.247 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-10-20T03:35:52.247 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2023-10-20T03:35:52.248 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2023-10-20T03:35:52.248 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 1, 2023-10-20T03:35:52.248 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 1, 2023-10-20T03:35:52.248 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 1, 2023-10-20T03:35:52.248 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 1, 2023-10-20T03:35:52.249 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 1, 2023-10-20T03:35:52.249 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2023-10-20T03:35:52.249 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2023-10-20T03:35:52.249 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2023-10-20T03:35:52.249 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2023-10-20T03:35:52.250 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2023-10-20T03:35:52.250 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2023-10-20T03:35:52.250 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2023-10-20T03:35:52.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.251 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "25", 2023-10-20T03:35:52.251 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2023-10-20T03:35:52.251 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2023-10-20T03:35:52.251 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2023-10-20T03:35:52.251 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2023-10-20T03:35:52.252 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2023-10-20T03:35:52.252 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2023-10-20T03:35:52.252 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2023-10-20T03:35:52.252 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2023-10-20T03:35:52.253 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2023-10-20T03:35:52.253 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2023-10-20T03:35:52.253 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2023-10-20T03:35:52.253 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2023-10-20T03:35:52.253 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2023-10-20T03:35:52.254 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2023-10-20T03:35:52.254 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2023-10-20T03:35:52.254 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2023-10-20T03:35:52.254 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2023-10-20T03:35:52.255 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2023-10-20T03:35:52.255 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-10-20T03:35:52.255 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-10-20T03:35:52.255 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2023-10-20T03:35:52.255 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2023-10-20T03:35:52.256 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2023-10-20T03:35:52.256 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2023-10-20T03:35:52.256 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2023-10-20T03:35:52.256 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2023-10-20T03:35:52.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.257 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2023-10-20T03:35:52.257 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2023-10-20T03:35:52.257 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2023-10-20T03:35:52.257 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2023-10-20T03:35:52.257 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2023-10-20T03:35:52.258 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2023-10-20T03:35:52.258 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2023-10-20T03:35:52.258 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2023-10-20T03:35:52.258 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2023-10-20T03:35:52.258 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2023-10-20T03:35:52.259 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2023-10-20T03:35:52.259 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2023-10-20T03:35:52.259 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_max": 32, 2023-10-20T03:35:52.259 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_min": 1 2023-10-20T03:35:52.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.260 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2023-10-20T03:35:52.260 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr": {} 2023-10-20T03:35:52.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.260 INFO:teuthology.orchestra.run.smithi033.stdout: "read_balance": { 2023-10-20T03:35:52.261 INFO:teuthology.orchestra.run.smithi033.stdout: "score_acting": 16.670000076293945, 2023-10-20T03:35:52.261 INFO:teuthology.orchestra.run.smithi033.stdout: "score_stable": 16.670000076293945, 2023-10-20T03:35:52.261 INFO:teuthology.orchestra.run.smithi033.stdout: "optimal_score": 0.059999998658895493, 2023-10-20T03:35:52.261 INFO:teuthology.orchestra.run.smithi033.stdout: "raw_score_acting": 1, 2023-10-20T03:35:52.261 INFO:teuthology.orchestra.run.smithi033.stdout: "raw_score_stable": 1, 2023-10-20T03:35:52.261 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity_weighted": 0.93946552276611328, 2023-10-20T03:35:52.262 INFO:teuthology.orchestra.run.smithi033.stdout: "average_primary_affinity": 0.93999999761581421, 2023-10-20T03:35:52.262 INFO:teuthology.orchestra.run.smithi033.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2023-10-20T03:35:52.262 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.262 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.263 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.263 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 2, 2023-10-20T03:35:52.263 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "rbd", 2023-10-20T03:35:52.263 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2023-10-20T03:35:03.136216+0000", 2023-10-20T03:35:52.263 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2023-10-20T03:35:52.264 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2023-10-20T03:35:52.264 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2023-10-20T03:35:52.264 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 2, 2023-10-20T03:35:52.264 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 1, 2023-10-20T03:35:52.264 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2023-10-20T03:35:52.265 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2023-10-20T03:35:52.265 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2023-10-20T03:35:52.265 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2023-10-20T03:35:52.265 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-10-20T03:35:52.266 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2023-10-20T03:35:52.266 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2023-10-20T03:35:52.266 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 8, 2023-10-20T03:35:52.266 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 8, 2023-10-20T03:35:52.266 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 8, 2023-10-20T03:35:52.267 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 8, 2023-10-20T03:35:52.267 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 8, 2023-10-20T03:35:52.267 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2023-10-20T03:35:52.267 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2023-10-20T03:35:52.267 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2023-10-20T03:35:52.268 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2023-10-20T03:35:52.268 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2023-10-20T03:35:52.268 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2023-10-20T03:35:52.268 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2023-10-20T03:35:52.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.269 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "28", 2023-10-20T03:35:52.269 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2023-10-20T03:35:52.269 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2023-10-20T03:35:52.269 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2023-10-20T03:35:52.270 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2023-10-20T03:35:52.270 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2023-10-20T03:35:52.270 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2023-10-20T03:35:52.270 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2023-10-20T03:35:52.270 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2023-10-20T03:35:52.271 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2023-10-20T03:35:52.271 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2023-10-20T03:35:52.271 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2023-10-20T03:35:52.271 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2023-10-20T03:35:52.272 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2023-10-20T03:35:52.272 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2023-10-20T03:35:52.272 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2023-10-20T03:35:52.272 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2023-10-20T03:35:52.272 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2023-10-20T03:35:52.273 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2023-10-20T03:35:52.273 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-10-20T03:35:52.273 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-10-20T03:35:52.273 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2023-10-20T03:35:52.273 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2023-10-20T03:35:52.274 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2023-10-20T03:35:52.274 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2023-10-20T03:35:52.274 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2023-10-20T03:35:52.274 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2023-10-20T03:35:52.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.275 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2023-10-20T03:35:52.275 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2023-10-20T03:35:52.275 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2023-10-20T03:35:52.276 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2023-10-20T03:35:52.276 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2023-10-20T03:35:52.276 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2023-10-20T03:35:52.276 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2023-10-20T03:35:52.277 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2023-10-20T03:35:52.277 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2023-10-20T03:35:52.277 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2023-10-20T03:35:52.277 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2023-10-20T03:35:52.277 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2023-10-20T03:35:52.278 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2023-10-20T03:35:52.278 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd": {} 2023-10-20T03:35:52.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.278 INFO:teuthology.orchestra.run.smithi033.stdout: "read_balance": { 2023-10-20T03:35:52.279 INFO:teuthology.orchestra.run.smithi033.stdout: "score_acting": 5.9499998092651367, 2023-10-20T03:35:52.279 INFO:teuthology.orchestra.run.smithi033.stdout: "score_stable": 5.9499998092651367, 2023-10-20T03:35:52.279 INFO:teuthology.orchestra.run.smithi033.stdout: "optimal_score": 0.62999999523162842, 2023-10-20T03:35:52.279 INFO:teuthology.orchestra.run.smithi033.stdout: "raw_score_acting": 3.75, 2023-10-20T03:35:52.279 INFO:teuthology.orchestra.run.smithi033.stdout: "raw_score_stable": 3.75, 2023-10-20T03:35:52.280 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity_weighted": 0.93946552276611328, 2023-10-20T03:35:52.280 INFO:teuthology.orchestra.run.smithi033.stdout: "average_primary_affinity": 0.93999999761581421, 2023-10-20T03:35:52.280 INFO:teuthology.orchestra.run.smithi033.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2023-10-20T03:35:52.280 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.281 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 3, 2023-10-20T03:35:52.281 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "unique_pool_0", 2023-10-20T03:35:52.281 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2023-10-20T03:35:12.214840+0000", 2023-10-20T03:35:52.281 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 8193, 2023-10-20T03:35:52.282 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2023-10-20T03:35:52.282 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 3, 2023-10-20T03:35:52.282 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 7, 2023-10-20T03:35:52.282 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 5, 2023-10-20T03:35:52.282 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 1, 2023-10-20T03:35:52.283 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2023-10-20T03:35:52.283 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2023-10-20T03:35:52.283 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2023-10-20T03:35:52.283 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-10-20T03:35:52.284 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2023-10-20T03:35:52.284 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2023-10-20T03:35:52.284 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 16, 2023-10-20T03:35:52.284 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 16, 2023-10-20T03:35:52.285 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 16, 2023-10-20T03:35:52.285 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 16, 2023-10-20T03:35:52.285 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 16, 2023-10-20T03:35:52.285 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2023-10-20T03:35:52.285 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2023-10-20T03:35:52.286 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2023-10-20T03:35:52.286 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2023-10-20T03:35:52.286 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2023-10-20T03:35:52.286 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2023-10-20T03:35:52.286 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2023-10-20T03:35:52.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.287 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "70", 2023-10-20T03:35:52.287 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2023-10-20T03:35:52.287 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2023-10-20T03:35:52.288 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2023-10-20T03:35:52.288 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2023-10-20T03:35:52.288 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2023-10-20T03:35:52.288 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 36, 2023-10-20T03:35:52.288 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 70, 2023-10-20T03:35:52.289 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2023-10-20T03:35:52.289 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2023-10-20T03:35:52.289 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2023-10-20T03:35:52.289 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2023-10-20T03:35:52.290 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2023-10-20T03:35:52.290 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2023-10-20T03:35:52.290 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2023-10-20T03:35:52.290 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2023-10-20T03:35:52.290 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2023-10-20T03:35:52.291 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2023-10-20T03:35:52.291 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2023-10-20T03:35:52.291 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-10-20T03:35:52.291 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-10-20T03:35:52.292 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2023-10-20T03:35:52.292 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2023-10-20T03:35:52.292 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2023-10-20T03:35:52.292 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "shecprofile", 2023-10-20T03:35:52.292 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2023-10-20T03:35:52.293 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2023-10-20T03:35:52.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.293 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2023-10-20T03:35:52.293 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2023-10-20T03:35:52.293 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2023-10-20T03:35:52.294 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2023-10-20T03:35:52.294 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2023-10-20T03:35:52.294 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2023-10-20T03:35:52.294 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2023-10-20T03:35:52.295 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2023-10-20T03:35:52.295 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 16384, 2023-10-20T03:35:52.295 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2023-10-20T03:35:52.295 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2023-10-20T03:35:52.295 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2023-10-20T03:35:52.296 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2023-10-20T03:35:52.296 INFO:teuthology.orchestra.run.smithi033.stdout: "rados": {} 2023-10-20T03:35:52.296 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.296 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.296 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-10-20T03:35:52.297 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2023-10-20T03:35:52.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.297 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2023-10-20T03:35:52.297 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "4872451c-1bc7-4f21-a95f-986fa428ea65", 2023-10-20T03:35:52.297 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.298 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.298 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.298 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0.072998046875, 2023-10-20T03:35:52.298 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.298 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.299 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.299 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 55, 2023-10-20T03:35:52.299 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.299 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.299 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.300 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.300 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.300 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6824", 2023-10-20T03:35:52.300 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:35:52.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.301 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.301 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6825", 2023-10-20T03:35:52.302 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:35:52.302 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.302 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.302 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.303 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.303 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.303 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6826", 2023-10-20T03:35:52.304 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:35:52.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.304 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.304 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6827", 2023-10-20T03:35:52.305 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:35:52.305 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.305 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.306 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.306 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.306 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.306 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6830", 2023-10-20T03:35:52.307 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:35:52.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.307 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.307 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6831", 2023-10-20T03:35:52.308 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:35:52.308 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.308 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.308 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.309 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.309 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.309 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6828", 2023-10-20T03:35:52.309 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:35:52.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.310 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.310 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6829", 2023-10-20T03:35:52.311 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:35:52.311 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.311 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.311 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6825/121544157", 2023-10-20T03:35:52.312 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6827/121544157", 2023-10-20T03:35:52.312 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6831/121544157", 2023-10-20T03:35:52.312 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6829/121544157", 2023-10-20T03:35:52.312 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.313 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.313 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.313 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.314 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2023-10-20T03:35:52.314 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "0b570426-e165-4caa-a914-e1bed7d805fc", 2023-10-20T03:35:52.314 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.314 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.315 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.315 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.315 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.315 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.316 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.316 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 56, 2023-10-20T03:35:52.316 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.316 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.317 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.317 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.317 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.317 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6808", 2023-10-20T03:35:52.318 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:35:52.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.318 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.319 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6809", 2023-10-20T03:35:52.319 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:35:52.319 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.319 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.319 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.320 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.320 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.320 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.320 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6810", 2023-10-20T03:35:52.321 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:35:52.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.321 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.321 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6811", 2023-10-20T03:35:52.322 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:35:52.322 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.322 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.322 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.323 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.323 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.323 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6818", 2023-10-20T03:35:52.324 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:35:52.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.324 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.324 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6820", 2023-10-20T03:35:52.325 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:35:52.325 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.325 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.325 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.326 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.326 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.326 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6813", 2023-10-20T03:35:52.327 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:35:52.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.327 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.327 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6815", 2023-10-20T03:35:52.328 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:35:52.328 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.328 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.328 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.40:6809/562043184", 2023-10-20T03:35:52.329 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.40:6811/562043184", 2023-10-20T03:35:52.329 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.40:6820/562043184", 2023-10-20T03:35:52.329 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.40:6815/562043184", 2023-10-20T03:35:52.329 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.329 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.330 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.330 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.331 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2023-10-20T03:35:52.331 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "e0184c10-844b-4bcb-adaf-a0d407b708ee", 2023-10-20T03:35:52.331 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.331 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.331 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.332 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.332 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.332 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.332 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.333 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2023-10-20T03:35:52.333 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.333 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.333 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.333 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.334 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.334 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6824", 2023-10-20T03:35:52.334 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:35:52.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.335 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.335 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.335 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6825", 2023-10-20T03:35:52.335 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:35:52.336 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.336 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.336 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.336 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.337 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.337 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.337 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6826", 2023-10-20T03:35:52.337 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:35:52.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.338 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.338 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6827", 2023-10-20T03:35:52.338 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:35:52.338 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.339 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.339 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.339 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.340 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.340 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6830", 2023-10-20T03:35:52.340 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:35:52.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.341 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.341 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6831", 2023-10-20T03:35:52.341 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:35:52.341 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.341 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.342 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.342 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.342 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.343 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6828", 2023-10-20T03:35:52.343 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:35:52.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.343 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.344 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6829", 2023-10-20T03:35:52.344 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:35:52.344 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.344 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.345 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.64:6825/3785405088", 2023-10-20T03:35:52.345 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.64:6827/3785405088", 2023-10-20T03:35:52.345 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.64:6831/3785405088", 2023-10-20T03:35:52.345 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.64:6829/3785405088", 2023-10-20T03:35:52.346 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.346 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.346 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.346 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.347 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2023-10-20T03:35:52.347 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "8120023b-1d48-411d-a642-2529a697b5c9", 2023-10-20T03:35:52.347 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.348 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.348 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.348 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0.9584503173828125, 2023-10-20T03:35:52.348 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.348 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.349 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.349 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 47, 2023-10-20T03:35:52.349 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.349 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.349 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.350 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.350 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.350 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.350 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6818", 2023-10-20T03:35:52.350 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:35:52.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.351 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.351 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.351 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6820", 2023-10-20T03:35:52.352 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:35:52.352 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.352 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.352 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.352 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.353 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.353 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.353 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6821", 2023-10-20T03:35:52.354 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:35:52.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.354 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.354 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6823", 2023-10-20T03:35:52.355 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:35:52.355 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.355 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.356 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.356 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.356 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.356 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.356 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6829", 2023-10-20T03:35:52.357 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:35:52.357 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.357 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.357 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.357 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6831", 2023-10-20T03:35:52.358 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:35:52.358 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.358 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.358 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.359 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.359 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.359 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.359 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.359 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6825", 2023-10-20T03:35:52.360 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:35:52.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.360 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.361 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6827", 2023-10-20T03:35:52.361 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:35:52.361 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.361 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.362 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.170:6820/1426674535", 2023-10-20T03:35:52.362 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.170:6823/1426674535", 2023-10-20T03:35:52.362 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.170:6831/1426674535", 2023-10-20T03:35:52.362 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.170:6827/1426674535", 2023-10-20T03:35:52.362 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.363 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.363 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.363 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.363 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.364 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.364 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2023-10-20T03:35:52.364 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "549b3b39-41d5-4d9b-932e-a7b3e8cc0684", 2023-10-20T03:35:52.364 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.364 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.365 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.365 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.365 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.365 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.365 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.366 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 55, 2023-10-20T03:35:52.366 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.366 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.366 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.366 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.367 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.367 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6808", 2023-10-20T03:35:52.367 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:35:52.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.368 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.368 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.368 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6809", 2023-10-20T03:35:52.368 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:35:52.369 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.369 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.369 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.369 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.370 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.370 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6810", 2023-10-20T03:35:52.370 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:35:52.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.371 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.371 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6811", 2023-10-20T03:35:52.371 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:35:52.372 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.372 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.372 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.372 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.373 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.373 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6814", 2023-10-20T03:35:52.373 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:35:52.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.374 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.374 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.374 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6815", 2023-10-20T03:35:52.374 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:35:52.375 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.375 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.375 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.375 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.376 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.376 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6812", 2023-10-20T03:35:52.376 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:35:52.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.377 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.377 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6813", 2023-10-20T03:35:52.377 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:35:52.378 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.378 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.378 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6809/3186493546", 2023-10-20T03:35:52.378 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6811/3186493546", 2023-10-20T03:35:52.379 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6815/3186493546", 2023-10-20T03:35:52.379 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6813/3186493546", 2023-10-20T03:35:52.379 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.379 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.380 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.380 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.380 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2023-10-20T03:35:52.381 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "a99ea3db-6433-4312-83d1-64ba69f028a4", 2023-10-20T03:35:52.381 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.381 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.381 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.381 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.382 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.382 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.382 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 21, 2023-10-20T03:35:52.382 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2023-10-20T03:35:52.383 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.383 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.383 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.383 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.383 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.384 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.384 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6814", 2023-10-20T03:35:52.384 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:35:52.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.385 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.385 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.385 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6817", 2023-10-20T03:35:52.385 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:35:52.385 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.386 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.386 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.386 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.386 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.387 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.387 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6821", 2023-10-20T03:35:52.387 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:35:52.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.388 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.388 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.388 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6823", 2023-10-20T03:35:52.388 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:35:52.388 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.389 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.389 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.389 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.389 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.389 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.390 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.390 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6829", 2023-10-20T03:35:52.390 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:35:52.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.391 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.391 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.391 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6831", 2023-10-20T03:35:52.391 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:35:52.391 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.391 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.392 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.392 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.392 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.392 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.393 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6825", 2023-10-20T03:35:52.393 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:35:52.393 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.393 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.393 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.394 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6827", 2023-10-20T03:35:52.394 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:35:52.394 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.394 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.395 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.395 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.40:6817/1211670976", 2023-10-20T03:35:52.395 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.40:6823/1211670976", 2023-10-20T03:35:52.395 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.40:6831/1211670976", 2023-10-20T03:35:52.395 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.40:6827/1211670976", 2023-10-20T03:35:52.396 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.396 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.396 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.396 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.397 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.397 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2023-10-20T03:35:52.397 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "d2b38063-49eb-4f56-bb8d-411419b71b9b", 2023-10-20T03:35:52.397 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 0, 2023-10-20T03:35:52.398 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.398 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.398 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.398 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.398 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.399 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.399 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 23, 2023-10-20T03:35:52.399 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 29, 2023-10-20T03:35:52.399 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.399 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.400 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.400 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.400 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.400 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6808", 2023-10-20T03:35:52.400 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2603014448 2023-10-20T03:35:52.401 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.401 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.401 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.401 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6809", 2023-10-20T03:35:52.401 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2603014448 2023-10-20T03:35:52.402 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.402 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.402 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.402 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.402 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.403 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.403 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.403 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6810", 2023-10-20T03:35:52.403 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2603014448 2023-10-20T03:35:52.404 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.404 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.404 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.404 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6811", 2023-10-20T03:35:52.404 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2603014448 2023-10-20T03:35:52.405 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.405 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.405 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.405 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.405 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.406 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.406 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.406 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6814", 2023-10-20T03:35:52.406 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2603014448 2023-10-20T03:35:52.407 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.407 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.407 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.407 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6815", 2023-10-20T03:35:52.407 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2603014448 2023-10-20T03:35:52.408 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.408 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.408 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.409 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.409 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.409 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.409 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6812", 2023-10-20T03:35:52.409 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2603014448 2023-10-20T03:35:52.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.410 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.410 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.410 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6813", 2023-10-20T03:35:52.410 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2603014448 2023-10-20T03:35:52.411 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.411 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.411 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.64:6809/2603014448", 2023-10-20T03:35:52.411 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.64:6811/2603014448", 2023-10-20T03:35:52.412 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.64:6815/2603014448", 2023-10-20T03:35:52.412 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.64:6813/2603014448", 2023-10-20T03:35:52.412 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.412 INFO:teuthology.orchestra.run.smithi033.stdout: "exists" 2023-10-20T03:35:52.412 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.413 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.413 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.413 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2023-10-20T03:35:52.413 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "80e34f64-35dc-4a6d-9f7f-c2263c78350d", 2023-10-20T03:35:52.414 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.414 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.414 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.414 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.415 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.415 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.415 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.415 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 31, 2023-10-20T03:35:52.415 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.416 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.416 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.416 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.416 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.416 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.417 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6810", 2023-10-20T03:35:52.417 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:35:52.417 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.417 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.418 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.418 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6811", 2023-10-20T03:35:52.418 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:35:52.418 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.418 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.419 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.419 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.419 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.419 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.420 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.420 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6812", 2023-10-20T03:35:52.420 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:35:52.420 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.420 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.421 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.421 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6813", 2023-10-20T03:35:52.421 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:35:52.421 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.421 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.422 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.422 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.422 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.422 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.423 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.423 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6816", 2023-10-20T03:35:52.423 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:35:52.423 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.423 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.424 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.424 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6817", 2023-10-20T03:35:52.424 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:35:52.424 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.425 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.425 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.425 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.425 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.425 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.426 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.426 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6814", 2023-10-20T03:35:52.426 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:35:52.426 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.427 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.427 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.427 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6815", 2023-10-20T03:35:52.427 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:35:52.427 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.428 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.428 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.428 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.170:6811/2804292733", 2023-10-20T03:35:52.428 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.170:6813/2804292733", 2023-10-20T03:35:52.428 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.170:6817/2804292733", 2023-10-20T03:35:52.429 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.170:6815/2804292733", 2023-10-20T03:35:52.429 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.429 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.429 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.429 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.430 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.430 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.430 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 8, 2023-10-20T03:35:52.430 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "1cd10625-66a8-4488-b1e5-378cb0062107", 2023-10-20T03:35:52.431 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.431 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.431 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.431 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.431 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.432 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.432 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.432 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 31, 2023-10-20T03:35:52.432 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.432 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.433 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.433 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.433 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.433 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.433 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6800", 2023-10-20T03:35:52.434 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:35:52.434 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.434 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.434 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.435 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6801", 2023-10-20T03:35:52.435 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:35:52.435 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.435 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.435 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.436 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.436 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.436 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.436 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.437 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6802", 2023-10-20T03:35:52.437 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:35:52.437 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.437 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.437 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.438 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6803", 2023-10-20T03:35:52.438 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:35:52.438 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.438 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.439 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.439 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.439 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.439 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.439 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6806", 2023-10-20T03:35:52.440 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:35:52.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.440 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.440 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.440 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6807", 2023-10-20T03:35:52.441 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:35:52.441 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.441 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.441 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.441 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.442 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.442 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.442 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.442 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6804", 2023-10-20T03:35:52.442 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:35:52.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.443 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.443 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.443 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6805", 2023-10-20T03:35:52.444 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:35:52.444 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.444 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.444 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.445 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6801/1638010589", 2023-10-20T03:35:52.445 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6803/1638010589", 2023-10-20T03:35:52.445 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6807/1638010589", 2023-10-20T03:35:52.445 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6805/1638010589", 2023-10-20T03:35:52.445 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.446 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.446 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.446 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.447 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.447 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 9, 2023-10-20T03:35:52.447 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4", 2023-10-20T03:35:52.447 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.447 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.448 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.448 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.448 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.448 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.449 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.449 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 56, 2023-10-20T03:35:52.449 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.449 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.449 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.449 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.450 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.450 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.450 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6800", 2023-10-20T03:35:52.450 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:35:52.451 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.451 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.451 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.451 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6801", 2023-10-20T03:35:52.451 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:35:52.452 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.452 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.452 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.452 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.452 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.453 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.453 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.453 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6802", 2023-10-20T03:35:52.453 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:35:52.453 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.454 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.454 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.454 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6803", 2023-10-20T03:35:52.454 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:35:52.455 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.455 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.455 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.455 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.456 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.456 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6806", 2023-10-20T03:35:52.456 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:35:52.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.457 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.457 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6807", 2023-10-20T03:35:52.457 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:35:52.457 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.458 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.458 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.458 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.459 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.459 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6804", 2023-10-20T03:35:52.459 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:35:52.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.459 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.460 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.460 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6805", 2023-10-20T03:35:52.460 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:35:52.460 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.460 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.461 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.40:6801/2173797306", 2023-10-20T03:35:52.461 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.40:6803/2173797306", 2023-10-20T03:35:52.461 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.40:6807/2173797306", 2023-10-20T03:35:52.461 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.40:6805/2173797306", 2023-10-20T03:35:52.461 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.462 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.462 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.462 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.463 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 10, 2023-10-20T03:35:52.463 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "42a67bcc-5c07-499d-8902-08c2f15e7fbb", 2023-10-20T03:35:52.463 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.464 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.464 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.464 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.464 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.464 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.465 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.465 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 56, 2023-10-20T03:35:52.465 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.465 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.465 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.466 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.466 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.466 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6800", 2023-10-20T03:35:52.467 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:35:52.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.467 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.467 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6801", 2023-10-20T03:35:52.468 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:35:52.468 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.468 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.468 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.469 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.469 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.469 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6802", 2023-10-20T03:35:52.469 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:35:52.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.470 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.470 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6803", 2023-10-20T03:35:52.471 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:35:52.471 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.471 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.471 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.472 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.472 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.472 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6806", 2023-10-20T03:35:52.472 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:35:52.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.473 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.473 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6807", 2023-10-20T03:35:52.474 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:35:52.474 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.474 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.474 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.475 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.475 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.475 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6804", 2023-10-20T03:35:52.476 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:35:52.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.476 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.476 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6805", 2023-10-20T03:35:52.477 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:35:52.477 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.477 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.477 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.64:6801/91701057", 2023-10-20T03:35:52.478 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.64:6803/91701057", 2023-10-20T03:35:52.478 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.64:6807/91701057", 2023-10-20T03:35:52.478 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.64:6805/91701057", 2023-10-20T03:35:52.478 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.479 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.479 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.479 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.480 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 11, 2023-10-20T03:35:52.480 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "ab1db647-aada-4deb-abdc-0a1baff548db", 2023-10-20T03:35:52.480 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.480 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.481 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.481 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.481 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.481 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.481 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.482 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 31, 2023-10-20T03:35:52.482 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.482 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.482 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.483 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.483 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.483 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6802", 2023-10-20T03:35:52.483 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:35:52.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.484 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.484 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6803", 2023-10-20T03:35:52.484 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:35:52.485 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.485 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.485 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.485 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.486 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.486 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6804", 2023-10-20T03:35:52.486 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:35:52.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.487 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.487 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6805", 2023-10-20T03:35:52.487 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:35:52.488 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.488 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.488 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.488 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.489 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.489 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6808", 2023-10-20T03:35:52.489 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:35:52.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.490 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.490 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6809", 2023-10-20T03:35:52.490 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:35:52.491 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.491 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.491 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.491 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.492 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.492 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6806", 2023-10-20T03:35:52.492 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:35:52.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.493 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.493 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6807", 2023-10-20T03:35:52.493 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:35:52.494 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.494 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.494 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.170:6803/1717984751", 2023-10-20T03:35:52.494 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.170:6805/1717984751", 2023-10-20T03:35:52.495 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.170:6809/1717984751", 2023-10-20T03:35:52.495 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.170:6807/1717984751", 2023-10-20T03:35:52.495 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.495 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.495 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.495 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.496 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 12, 2023-10-20T03:35:52.496 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "b20cee74-2cce-4153-97c2-56e9db40c712", 2023-10-20T03:35:52.496 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.497 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2023-10-20T03:35:52.497 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2023-10-20T03:35:52.497 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.497 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.497 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.498 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.498 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 31, 2023-10-20T03:35:52.498 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.498 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.498 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.499 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.499 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.499 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6816", 2023-10-20T03:35:52.499 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:35:52.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.500 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.500 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6817", 2023-10-20T03:35:52.500 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:35:52.500 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.501 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.501 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.501 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.502 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.502 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6818", 2023-10-20T03:35:52.502 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:35:52.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.503 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.503 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6819", 2023-10-20T03:35:52.503 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:35:52.503 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.504 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.504 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.504 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.505 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.505 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6822", 2023-10-20T03:35:52.505 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:35:52.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.506 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.506 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6823", 2023-10-20T03:35:52.506 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:35:52.506 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.507 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.507 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.507 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.508 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.508 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6820", 2023-10-20T03:35:52.508 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:35:52.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.509 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.509 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6821", 2023-10-20T03:35:52.509 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:35:52.509 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.509 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.510 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6817/3797427065", 2023-10-20T03:35:52.510 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6819/3797427065", 2023-10-20T03:35:52.510 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6823/3797427065", 2023-10-20T03:35:52.511 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6821/3797427065", 2023-10-20T03:35:52.511 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.511 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.511 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.511 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.512 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.512 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 13, 2023-10-20T03:35:52.512 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "0974cd04-c690-4cba-ac05-5d404493dd1d", 2023-10-20T03:35:52.512 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.512 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.513 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.513 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.513 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.513 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.514 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.514 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 56, 2023-10-20T03:35:52.514 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.514 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.514 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.515 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.515 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.515 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6812", 2023-10-20T03:35:52.516 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:35:52.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.516 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.517 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6816", 2023-10-20T03:35:52.517 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:35:52.517 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.517 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.518 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.518 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.518 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.518 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6819", 2023-10-20T03:35:52.519 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:35:52.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.519 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.520 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6822", 2023-10-20T03:35:52.520 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:35:52.520 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.520 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.521 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.521 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.521 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.522 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6828", 2023-10-20T03:35:52.522 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:35:52.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.522 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.523 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6830", 2023-10-20T03:35:52.523 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:35:52.523 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.523 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.524 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.524 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.524 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.525 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6824", 2023-10-20T03:35:52.525 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:35:52.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.525 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.525 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.526 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6826", 2023-10-20T03:35:52.526 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:35:52.526 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.526 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.527 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.40:6816/3041424233", 2023-10-20T03:35:52.527 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.40:6822/3041424233", 2023-10-20T03:35:52.527 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.40:6830/3041424233", 2023-10-20T03:35:52.527 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.40:6826/3041424233", 2023-10-20T03:35:52.528 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.528 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.528 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.528 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.529 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 14, 2023-10-20T03:35:52.529 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "3bf53124-13b6-4e5c-9868-2382719ffe25", 2023-10-20T03:35:52.529 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.530 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.530 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.530 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.530 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.531 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.531 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.531 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 31, 2023-10-20T03:35:52.531 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.532 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.532 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.532 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.532 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.533 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6816", 2023-10-20T03:35:52.533 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:35:52.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.534 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.534 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6817", 2023-10-20T03:35:52.534 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:35:52.534 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.534 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.535 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.535 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.535 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.536 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6818", 2023-10-20T03:35:52.536 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:35:52.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.537 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.537 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6819", 2023-10-20T03:35:52.537 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:35:52.537 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.537 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.538 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.538 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.538 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.539 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6822", 2023-10-20T03:35:52.539 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:35:52.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.539 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.540 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6823", 2023-10-20T03:35:52.540 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:35:52.540 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.540 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.541 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.541 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.541 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.542 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6820", 2023-10-20T03:35:52.542 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:35:52.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.542 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.543 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6821", 2023-10-20T03:35:52.543 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:35:52.543 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.543 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.544 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.64:6817/2940007286", 2023-10-20T03:35:52.544 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.64:6819/2940007286", 2023-10-20T03:35:52.544 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.64:6823/2940007286", 2023-10-20T03:35:52.544 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.64:6821/2940007286", 2023-10-20T03:35:52.544 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.545 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.545 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.545 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.546 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 15, 2023-10-20T03:35:52.546 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "b0b5ffe5-a25b-412f-818f-b60907c0089b", 2023-10-20T03:35:52.546 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:35:52.546 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:35:52.547 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:35:52.547 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:35:52.547 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:35:52.547 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:35:52.547 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:35:52.548 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 29, 2023-10-20T03:35:52.548 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:35:52.548 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:35:52.548 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:35:52.548 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.549 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.549 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6819", 2023-10-20T03:35:52.549 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:35:52.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.550 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.550 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6822", 2023-10-20T03:35:52.550 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:35:52.551 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.551 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.551 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:35:52.551 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.552 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.552 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6824", 2023-10-20T03:35:52.552 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:35:52.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.553 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.553 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6826", 2023-10-20T03:35:52.553 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:35:52.553 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.554 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.554 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:35:52.554 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.555 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.555 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6832", 2023-10-20T03:35:52.555 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:35:52.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.556 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.556 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6833", 2023-10-20T03:35:52.556 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:35:52.556 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.557 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.557 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:35:52.557 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:35:52.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.558 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:35:52.558 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6828", 2023-10-20T03:35:52.558 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:35:52.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.559 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:35:52.559 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6830", 2023-10-20T03:35:52.559 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:35:52.559 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.560 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.560 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.170:6822/826332016", 2023-10-20T03:35:52.560 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.170:6826/826332016", 2023-10-20T03:35:52.561 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.170:6833/826332016", 2023-10-20T03:35:52.561 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.170:6830/826332016", 2023-10-20T03:35:52.561 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:35:52.561 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:35:52.561 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:35:52.562 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.562 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.562 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-10-20T03:35:52.562 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_xinfo": [ 2023-10-20T03:35:52.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.563 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2023-10-20T03:35:52.563 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.563 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.563 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.564 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.564 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.564 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.204996+0000", 2023-10-20T03:35:52.564 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.565 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2023-10-20T03:35:52.565 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.565 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.565 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.566 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.566 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.566 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.164818+0000", 2023-10-20T03:35:52.566 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.567 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2023-10-20T03:35:52.567 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.567 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.568 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.568 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.568 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.568 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.304156+0000", 2023-10-20T03:35:52.568 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.569 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2023-10-20T03:35:52.569 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.570 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.570 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.570 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.570 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.570 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.175698+0000", 2023-10-20T03:35:52.571 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.571 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2023-10-20T03:35:52.572 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.572 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.572 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.572 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.572 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.573 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.169661+0000", 2023-10-20T03:35:52.573 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.573 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2023-10-20T03:35:52.574 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.574 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.574 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.574 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.575 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.575 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:57.863151+0000", 2023-10-20T03:35:52.575 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.576 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2023-10-20T03:35:52.576 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.576 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.576 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.577 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.577 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.577 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.279628+0000", 2023-10-20T03:35:52.577 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 29 2023-10-20T03:35:52.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.578 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2023-10-20T03:35:52.578 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.578 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.578 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.579 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.579 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.579 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.238525+0000", 2023-10-20T03:35:52.579 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.580 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 8, 2023-10-20T03:35:52.580 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.580 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.580 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.581 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.581 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.581 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.074220+0000", 2023-10-20T03:35:52.581 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.582 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 9, 2023-10-20T03:35:52.582 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.582 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.583 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.583 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.583 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.583 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.020311+0000", 2023-10-20T03:35:52.583 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.584 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.584 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 10, 2023-10-20T03:35:52.584 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.584 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.585 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.585 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.585 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.585 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.177070+0000", 2023-10-20T03:35:52.586 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.586 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.586 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 11, 2023-10-20T03:35:52.586 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.587 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.587 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.587 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.587 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.587 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.063374+0000", 2023-10-20T03:35:52.588 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.588 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 12, 2023-10-20T03:35:52.588 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.589 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.589 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.589 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.589 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2023-10-20T03:35:52.589 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.069832+0000", 2023-10-20T03:35:52.590 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.590 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 13, 2023-10-20T03:35:52.590 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.591 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.591 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.591 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.591 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.591 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.130292+0000", 2023-10-20T03:35:52.592 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.592 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.592 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.592 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 14, 2023-10-20T03:35:52.592 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.593 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.593 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.593 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.593 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.594 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.265757+0000", 2023-10-20T03:35:52.594 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.594 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.594 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 15, 2023-10-20T03:35:52.595 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:35:52.595 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:35:52.595 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:35:52.595 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:35:52.595 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:35:52.596 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.317166+0000", 2023-10-20T03:35:52.596 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:35:52.596 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.596 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-10-20T03:35:52.596 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap": [], 2023-10-20T03:35:52.597 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap_items": [], 2023-10-20T03:35:52.597 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap_primaries": [], 2023-10-20T03:35:52.597 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_temp": [], 2023-10-20T03:35:52.597 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_temp": [], 2023-10-20T03:35:52.597 INFO:teuthology.orchestra.run.smithi033.stdout: "blocklist": {}, 2023-10-20T03:35:52.598 INFO:teuthology.orchestra.run.smithi033.stdout: "range_blocklist": {}, 2023-10-20T03:35:52.598 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profiles": { 2023-10-20T03:35:52.598 INFO:teuthology.orchestra.run.smithi033.stdout: "default": { 2023-10-20T03:35:52.598 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-failure-domain": "osd", 2023-10-20T03:35:52.598 INFO:teuthology.orchestra.run.smithi033.stdout: "k": "2", 2023-10-20T03:35:52.599 INFO:teuthology.orchestra.run.smithi033.stdout: "m": "1", 2023-10-20T03:35:52.599 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin": "jerasure", 2023-10-20T03:35:52.599 INFO:teuthology.orchestra.run.smithi033.stdout: "technique": "reed_sol_van" 2023-10-20T03:35:52.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.599 INFO:teuthology.orchestra.run.smithi033.stdout: "shecprofile": { 2023-10-20T03:35:52.600 INFO:teuthology.orchestra.run.smithi033.stdout: "c": "2", 2023-10-20T03:35:52.600 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-device-class": "", 2023-10-20T03:35:52.600 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-failure-domain": "osd", 2023-10-20T03:35:52.600 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-root": "default", 2023-10-20T03:35:52.600 INFO:teuthology.orchestra.run.smithi033.stdout: "k": "4", 2023-10-20T03:35:52.600 INFO:teuthology.orchestra.run.smithi033.stdout: "m": "3", 2023-10-20T03:35:52.601 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "shecprofile", 2023-10-20T03:35:52.601 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin": "shec", 2023-10-20T03:35:52.601 INFO:teuthology.orchestra.run.smithi033.stdout: "technique": "multiple" 2023-10-20T03:35:52.601 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.602 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps_queue": [ 2023-10-20T03:35:52.602 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.602 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 3, 2023-10-20T03:35:52.602 INFO:teuthology.orchestra.run.smithi033.stdout: "snaps": [ 2023-10-20T03:35:52.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.603 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 2, 2023-10-20T03:35:52.603 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 3 2023-10-20T03:35:52.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.604 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 6, 2023-10-20T03:35:52.604 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 2 2023-10-20T03:35:52.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.604 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.604 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 9, 2023-10-20T03:35:52.605 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 5 2023-10-20T03:35:52.605 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.605 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.605 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 15, 2023-10-20T03:35:52.605 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:35:52.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.606 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 22, 2023-10-20T03:35:52.606 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:35:52.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.607 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 25, 2023-10-20T03:35:52.607 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:35:52.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.608 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 28, 2023-10-20T03:35:52.608 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 3 2023-10-20T03:35:52.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.609 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 33, 2023-10-20T03:35:52.609 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:35:52.609 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.609 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.609 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.610 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-10-20T03:35:52.610 INFO:teuthology.orchestra.run.smithi033.stdout: "new_removed_snaps": [ 2023-10-20T03:35:52.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.610 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 3, 2023-10-20T03:35:52.610 INFO:teuthology.orchestra.run.smithi033.stdout: "snaps": [ 2023-10-20T03:35:52.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.611 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 33, 2023-10-20T03:35:52.611 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:35:52.611 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.611 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.612 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.612 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-10-20T03:35:52.612 INFO:teuthology.orchestra.run.smithi033.stdout: "new_purged_snaps": [], 2023-10-20T03:35:52.612 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_node_flags": {}, 2023-10-20T03:35:52.612 INFO:teuthology.orchestra.run.smithi033.stdout: "device_class_flags": {}, 2023-10-20T03:35:52.613 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode": { 2023-10-20T03:35:52.613 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_enabled": false, 2023-10-20T03:35:52.613 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_bucket_count": 0, 2023-10-20T03:35:52.613 INFO:teuthology.orchestra.run.smithi033.stdout: "degraded_stretch_mode": 0, 2023-10-20T03:35:52.614 INFO:teuthology.orchestra.run.smithi033.stdout: "recovering_stretch_mode": 0, 2023-10-20T03:35:52.614 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_bucket": 0 2023-10-20T03:35:52.614 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.614 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:52.615 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:35:52.616 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:52.616+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:52.618 INFO:tasks.thrashosds.thrasher:j is {'epoch': 70, 'fsid': 'b920f165-e9e7-4c3e-8d7b-cfdc002ae589', 'created': '2023-10-20T03:34:47.883622+0000', 'modified': '2023-10-20T03:35:51.927563+0000', 'last_up_change': '2023-10-20T03:35:11.150103+0000', 'last_in_change': '2023-10-20T03:35:35.398012+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-20T03:35:00.991869+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 16.670000076293945, 'score_stable': 16.670000076293945, 'optimal_score': 0.05999999865889549, 'raw_score_acting': 1, 'raw_score_stable': 1, 'primary_affinity_weighted': 0.9394655227661133, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-20T03:35:03.136216+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 5.949999809265137, 'score_stable': 5.949999809265137, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 3.75, 'raw_score_stable': 3.75, 'primary_affinity_weighted': 0.9394655227661133, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2023-10-20T03:35:12.214840+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 7, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 36, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'shecprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '4872451c-1bc7-4f21-a95f-986fa428ea65', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.072998046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 55, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6824', 'nonce': 121544157}, {'type': 'v1', 'addr': '172.21.15.33:6825', 'nonce': 121544157}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6826', 'nonce': 121544157}, {'type': 'v1', 'addr': '172.21.15.33:6827', 'nonce': 121544157}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6830', 'nonce': 121544157}, {'type': 'v1', 'addr': '172.21.15.33:6831', 'nonce': 121544157}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6828', 'nonce': 121544157}, {'type': 'v1', 'addr': '172.21.15.33:6829', 'nonce': 121544157}]}, 'public_addr': '172.21.15.33:6825/121544157', 'cluster_addr': '172.21.15.33:6827/121544157', 'heartbeat_back_addr': '172.21.15.33:6831/121544157', 'heartbeat_front_addr': '172.21.15.33:6829/121544157', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0b570426-e165-4caa-a914-e1bed7d805fc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6808', 'nonce': 562043184}, {'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 562043184}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6810', 'nonce': 562043184}, {'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 562043184}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6818', 'nonce': 562043184}, {'type': 'v1', 'addr': '172.21.15.40:6820', 'nonce': 562043184}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6813', 'nonce': 562043184}, {'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 562043184}]}, 'public_addr': '172.21.15.40:6809/562043184', 'cluster_addr': '172.21.15.40:6811/562043184', 'heartbeat_back_addr': '172.21.15.40:6820/562043184', 'heartbeat_front_addr': '172.21.15.40:6815/562043184', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e0184c10-844b-4bcb-adaf-a0d407b708ee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6824', 'nonce': 3785405088}, {'type': 'v1', 'addr': '172.21.15.64:6825', 'nonce': 3785405088}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6826', 'nonce': 3785405088}, {'type': 'v1', 'addr': '172.21.15.64:6827', 'nonce': 3785405088}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6830', 'nonce': 3785405088}, {'type': 'v1', 'addr': '172.21.15.64:6831', 'nonce': 3785405088}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6828', 'nonce': 3785405088}, {'type': 'v1', 'addr': '172.21.15.64:6829', 'nonce': 3785405088}]}, 'public_addr': '172.21.15.64:6825/3785405088', 'cluster_addr': '172.21.15.64:6827/3785405088', 'heartbeat_back_addr': '172.21.15.64:6831/3785405088', 'heartbeat_front_addr': '172.21.15.64:6829/3785405088', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8120023b-1d48-411d-a642-2529a697b5c9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9584503173828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6818', 'nonce': 1426674535}, {'type': 'v1', 'addr': '172.21.15.170:6820', 'nonce': 1426674535}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6821', 'nonce': 1426674535}, {'type': 'v1', 'addr': '172.21.15.170:6823', 'nonce': 1426674535}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6829', 'nonce': 1426674535}, {'type': 'v1', 'addr': '172.21.15.170:6831', 'nonce': 1426674535}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6825', 'nonce': 1426674535}, {'type': 'v1', 'addr': '172.21.15.170:6827', 'nonce': 1426674535}]}, 'public_addr': '172.21.15.170:6820/1426674535', 'cluster_addr': '172.21.15.170:6823/1426674535', 'heartbeat_back_addr': '172.21.15.170:6831/1426674535', 'heartbeat_front_addr': '172.21.15.170:6827/1426674535', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '549b3b39-41d5-4d9b-932e-a7b3e8cc0684', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 55, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6808', 'nonce': 3186493546}, {'type': 'v1', 'addr': '172.21.15.33:6809', 'nonce': 3186493546}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6810', 'nonce': 3186493546}, {'type': 'v1', 'addr': '172.21.15.33:6811', 'nonce': 3186493546}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6814', 'nonce': 3186493546}, {'type': 'v1', 'addr': '172.21.15.33:6815', 'nonce': 3186493546}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6812', 'nonce': 3186493546}, {'type': 'v1', 'addr': '172.21.15.33:6813', 'nonce': 3186493546}]}, 'public_addr': '172.21.15.33:6809/3186493546', 'cluster_addr': '172.21.15.33:6811/3186493546', 'heartbeat_back_addr': '172.21.15.33:6815/3186493546', 'heartbeat_front_addr': '172.21.15.33:6813/3186493546', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a99ea3db-6433-4312-83d1-64ba69f028a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6814', 'nonce': 1211670976}, {'type': 'v1', 'addr': '172.21.15.40:6817', 'nonce': 1211670976}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6821', 'nonce': 1211670976}, {'type': 'v1', 'addr': '172.21.15.40:6823', 'nonce': 1211670976}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6829', 'nonce': 1211670976}, {'type': 'v1', 'addr': '172.21.15.40:6831', 'nonce': 1211670976}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6825', 'nonce': 1211670976}, {'type': 'v1', 'addr': '172.21.15.40:6827', 'nonce': 1211670976}]}, 'public_addr': '172.21.15.40:6817/1211670976', 'cluster_addr': '172.21.15.40:6823/1211670976', 'heartbeat_back_addr': '172.21.15.40:6831/1211670976', 'heartbeat_front_addr': '172.21.15.40:6827/1211670976', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd2b38063-49eb-4f56-bb8d-411419b71b9b', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 23, 'down_at': 29, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 2603014448}, {'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 2603014448}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 2603014448}, {'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 2603014448}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 2603014448}, {'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 2603014448}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 2603014448}, {'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 2603014448}]}, 'public_addr': '172.21.15.64:6809/2603014448', 'cluster_addr': '172.21.15.64:6811/2603014448', 'heartbeat_back_addr': '172.21.15.64:6815/2603014448', 'heartbeat_front_addr': '172.21.15.64:6813/2603014448', 'state': ['exists']}, {'osd': 7, 'uuid': '80e34f64-35dc-4a6d-9f7f-c2263c78350d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6810', 'nonce': 2804292733}, {'type': 'v1', 'addr': '172.21.15.170:6811', 'nonce': 2804292733}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6812', 'nonce': 2804292733}, {'type': 'v1', 'addr': '172.21.15.170:6813', 'nonce': 2804292733}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6816', 'nonce': 2804292733}, {'type': 'v1', 'addr': '172.21.15.170:6817', 'nonce': 2804292733}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6814', 'nonce': 2804292733}, {'type': 'v1', 'addr': '172.21.15.170:6815', 'nonce': 2804292733}]}, 'public_addr': '172.21.15.170:6811/2804292733', 'cluster_addr': '172.21.15.170:6813/2804292733', 'heartbeat_back_addr': '172.21.15.170:6817/2804292733', 'heartbeat_front_addr': '172.21.15.170:6815/2804292733', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1cd10625-66a8-4488-b1e5-378cb0062107', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6800', 'nonce': 1638010589}, {'type': 'v1', 'addr': '172.21.15.33:6801', 'nonce': 1638010589}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6802', 'nonce': 1638010589}, {'type': 'v1', 'addr': '172.21.15.33:6803', 'nonce': 1638010589}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6806', 'nonce': 1638010589}, {'type': 'v1', 'addr': '172.21.15.33:6807', 'nonce': 1638010589}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6804', 'nonce': 1638010589}, {'type': 'v1', 'addr': '172.21.15.33:6805', 'nonce': 1638010589}]}, 'public_addr': '172.21.15.33:6801/1638010589', 'cluster_addr': '172.21.15.33:6803/1638010589', 'heartbeat_back_addr': '172.21.15.33:6807/1638010589', 'heartbeat_front_addr': '172.21.15.33:6805/1638010589', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6800', 'nonce': 2173797306}, {'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 2173797306}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6802', 'nonce': 2173797306}, {'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 2173797306}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6806', 'nonce': 2173797306}, {'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 2173797306}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6804', 'nonce': 2173797306}, {'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 2173797306}]}, 'public_addr': '172.21.15.40:6801/2173797306', 'cluster_addr': '172.21.15.40:6803/2173797306', 'heartbeat_back_addr': '172.21.15.40:6807/2173797306', 'heartbeat_front_addr': '172.21.15.40:6805/2173797306', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '42a67bcc-5c07-499d-8902-08c2f15e7fbb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 91701057}, {'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 91701057}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 91701057}, {'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 91701057}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 91701057}, {'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 91701057}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 91701057}, {'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 91701057}]}, 'public_addr': '172.21.15.64:6801/91701057', 'cluster_addr': '172.21.15.64:6803/91701057', 'heartbeat_back_addr': '172.21.15.64:6807/91701057', 'heartbeat_front_addr': '172.21.15.64:6805/91701057', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ab1db647-aada-4deb-abdc-0a1baff548db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6802', 'nonce': 1717984751}, {'type': 'v1', 'addr': '172.21.15.170:6803', 'nonce': 1717984751}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6804', 'nonce': 1717984751}, {'type': 'v1', 'addr': '172.21.15.170:6805', 'nonce': 1717984751}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6808', 'nonce': 1717984751}, {'type': 'v1', 'addr': '172.21.15.170:6809', 'nonce': 1717984751}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6806', 'nonce': 1717984751}, {'type': 'v1', 'addr': '172.21.15.170:6807', 'nonce': 1717984751}]}, 'public_addr': '172.21.15.170:6803/1717984751', 'cluster_addr': '172.21.15.170:6805/1717984751', 'heartbeat_back_addr': '172.21.15.170:6809/1717984751', 'heartbeat_front_addr': '172.21.15.170:6807/1717984751', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': 'b20cee74-2cce-4153-97c2-56e9db40c712', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6816', 'nonce': 3797427065}, {'type': 'v1', 'addr': '172.21.15.33:6817', 'nonce': 3797427065}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6818', 'nonce': 3797427065}, {'type': 'v1', 'addr': '172.21.15.33:6819', 'nonce': 3797427065}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6822', 'nonce': 3797427065}, {'type': 'v1', 'addr': '172.21.15.33:6823', 'nonce': 3797427065}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6820', 'nonce': 3797427065}, {'type': 'v1', 'addr': '172.21.15.33:6821', 'nonce': 3797427065}]}, 'public_addr': '172.21.15.33:6817/3797427065', 'cluster_addr': '172.21.15.33:6819/3797427065', 'heartbeat_back_addr': '172.21.15.33:6823/3797427065', 'heartbeat_front_addr': '172.21.15.33:6821/3797427065', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '0974cd04-c690-4cba-ac05-5d404493dd1d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6812', 'nonce': 3041424233}, {'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 3041424233}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6819', 'nonce': 3041424233}, {'type': 'v1', 'addr': '172.21.15.40:6822', 'nonce': 3041424233}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6828', 'nonce': 3041424233}, {'type': 'v1', 'addr': '172.21.15.40:6830', 'nonce': 3041424233}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6824', 'nonce': 3041424233}, {'type': 'v1', 'addr': '172.21.15.40:6826', 'nonce': 3041424233}]}, 'public_addr': '172.21.15.40:6816/3041424233', 'cluster_addr': '172.21.15.40:6822/3041424233', 'heartbeat_back_addr': '172.21.15.40:6830/3041424233', 'heartbeat_front_addr': '172.21.15.40:6826/3041424233', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '3bf53124-13b6-4e5c-9868-2382719ffe25', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6816', 'nonce': 2940007286}, {'type': 'v1', 'addr': '172.21.15.64:6817', 'nonce': 2940007286}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6818', 'nonce': 2940007286}, {'type': 'v1', 'addr': '172.21.15.64:6819', 'nonce': 2940007286}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6822', 'nonce': 2940007286}, {'type': 'v1', 'addr': '172.21.15.64:6823', 'nonce': 2940007286}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6820', 'nonce': 2940007286}, {'type': 'v1', 'addr': '172.21.15.64:6821', 'nonce': 2940007286}]}, 'public_addr': '172.21.15.64:6817/2940007286', 'cluster_addr': '172.21.15.64:6819/2940007286', 'heartbeat_back_addr': '172.21.15.64:6823/2940007286', 'heartbeat_front_addr': '172.21.15.64:6821/2940007286', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': 'b0b5ffe5-a25b-412f-818f-b60907c0089b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6819', 'nonce': 826332016}, {'type': 'v1', 'addr': '172.21.15.170:6822', 'nonce': 826332016}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6824', 'nonce': 826332016}, {'type': 'v1', 'addr': '172.21.15.170:6826', 'nonce': 826332016}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 826332016}, {'type': 'v1', 'addr': '172.21.15.170:6833', 'nonce': 826332016}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6828', 'nonce': 826332016}, {'type': 'v1', 'addr': '172.21.15.170:6830', 'nonce': 826332016}]}, 'public_addr': '172.21.15.170:6822/826332016', 'cluster_addr': '172.21.15.170:6826/826332016', 'heartbeat_back_addr': '172.21.15.170:6833/826332016', 'heartbeat_front_addr': '172.21.15.170:6830/826332016', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.204996+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.164818+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.304156+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.175698+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.169661+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:57.863151+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.279628+0000', 'dead_epoch': 29}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.238525+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.074220+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.020311+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.177070+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.063374+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.069832+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.130292+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.265757+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.317166+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'shecprofile': {'c': '2', 'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'k': '4', 'm': '3', 'name': 'shecprofile', 'plugin': 'shec', 'technique': 'multiple'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 3}, {'begin': 6, 'length': 2}, {'begin': 9, 'length': 5}, {'begin': 15, 'length': 1}, {'begin': 22, 'length': 1}, {'begin': 25, 'length': 1}, {'begin': 28, 'length': 3}, {'begin': 33, 'length': 1}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 33, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-10-20T03:35:52.619 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T03:35:52.620 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:52.620 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:52.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:52.621 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:35:52.621 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.621 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=21, reqid=client.4570.0:198, at_version=36'13, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:52.621 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:52.621 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 34.012602725000001, 2023-10-20T03:35:52.622 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0086318609999999994, 2023-10-20T03:35:52.622 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.622 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:52.622 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.623 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.623 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.623 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:52.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.624 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.624 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:52.624 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.625 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.266450+0000", 2023-10-20T03:35:52.625 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.58e-07 2023-10-20T03:35:52.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.626 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.626 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.267902+0000", 2023-10-20T03:35:52.626 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014514409999999999 2023-10-20T03:35:52.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.627 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.627 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.267916+0000", 2023-10-20T03:35:52.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4664999999999999e-05 2023-10-20T03:35:52.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.267931+0000", 2023-10-20T03:35:52.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4192999999999999e-05 2023-10-20T03:35:52.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.268007+0000", 2023-10-20T03:35:52.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6058999999999998e-05 2023-10-20T03:35:52.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.630 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.630 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.268017+0000", 2023-10-20T03:35:52.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0648999999999999e-05 2023-10-20T03:35:52.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.631 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275053+0000", 2023-10-20T03:35:52.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0070355699999999997 2023-10-20T03:35:52.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.275082+0000", 2023-10-20T03:35:52.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8926000000000001e-05 2023-10-20T03:35:52.633 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.633 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.634 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.634 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=22, reqid=client.4570.0:199, at_version=36'14, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:35:52.634 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:18.284006+0000", 2023-10-20T03:35:52.635 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 33.995046277999997, 2023-10-20T03:35:52.635 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0061959429999999998, 2023-10-20T03:35:52.635 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.635 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:52.636 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.636 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.636 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.636 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284006+0000", 2023-10-20T03:35:52.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.637 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284006+0000", 2023-10-20T03:35:52.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.639 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284008+0000", 2023-10-20T03:35:52.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.398e-06 2023-10-20T03:35:52.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.640 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284090+0000", 2023-10-20T03:35:52.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2686000000000001e-05 2023-10-20T03:35:52.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284111+0000", 2023-10-20T03:35:52.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0363000000000001e-05 2023-10-20T03:35:52.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.642 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284123+0000", 2023-10-20T03:35:52.642 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2213e-05 2023-10-20T03:35:52.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.643 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284222+0000", 2023-10-20T03:35:52.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8587999999999999e-05 2023-10-20T03:35:52.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.644 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.644 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.284237+0000", 2023-10-20T03:35:52.644 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.541e-05 2023-10-20T03:35:52.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.290159+0000", 2023-10-20T03:35:52.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0059216299999999998 2023-10-20T03:35:52.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.646 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.646 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:18.290202+0000", 2023-10-20T03:35:52.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3655000000000001e-05 2023-10-20T03:35:52.647 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.647 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.647 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.647 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.648 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:212 3.5s0 3:aed1dba0:::smithi17039300-29:2 [read 0~425984,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:52.648 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.277328+0000", 2023-10-20T03:35:52.648 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 33.001725127999997, 2023-10-20T03:35:52.648 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0085440599999999992, 2023-10-20T03:35:52.648 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.649 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:52.649 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:52.649 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:52.649 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:52.649 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 212 2023-10-20T03:35:52.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.650 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.651 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277328+0000", 2023-10-20T03:35:52.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.651 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.651 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.652 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277328+0000", 2023-10-20T03:35:52.652 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.652 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.653 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277330+0000", 2023-10-20T03:35:52.653 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7690000000000001e-06 2023-10-20T03:35:52.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.653 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.654 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.654 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277343+0000", 2023-10-20T03:35:52.654 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2221e-05 2023-10-20T03:35:52.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.654 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.655 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277368+0000", 2023-10-20T03:35:52.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5544000000000001e-05 2023-10-20T03:35:52.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.655 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277382+0000", 2023-10-20T03:35:52.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4246e-05 2023-10-20T03:35:52.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.657 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.657 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277510+0000", 2023-10-20T03:35:52.657 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012724599999999999 2023-10-20T03:35:52.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.658 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2023-10-20T03:35:52.658 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277524+0000", 2023-10-20T03:35:52.658 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4466e-05 2023-10-20T03:35:52.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.659 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.659 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.659 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.281643+0000", 2023-10-20T03:35:52.659 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041191680000000003 2023-10-20T03:35:52.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.660 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.660 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:52.660 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.281815+0000", 2023-10-20T03:35:52.660 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000171991 2023-10-20T03:35:52.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.661 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.661 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.285872+0000", 2023-10-20T03:35:52.661 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0040564090000000004 2023-10-20T03:35:52.662 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.662 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.662 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.663 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:213 3.5s0 3:aed1dba0:::smithi17039300-29:2 [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:52.663 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.277413+0000", 2023-10-20T03:35:52.663 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 33.001639255999997, 2023-10-20T03:35:52.663 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0087379769999999992, 2023-10-20T03:35:52.664 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.664 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:52.664 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:52.664 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:52.665 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:52.665 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 213 2023-10-20T03:35:52.665 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.665 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.665 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.666 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.666 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277413+0000", 2023-10-20T03:35:52.666 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.666 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.667 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.667 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.667 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277413+0000", 2023-10-20T03:35:52.667 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.667 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.668 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.668 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.668 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277414+0000", 2023-10-20T03:35:52.668 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.7599999999999997e-07 2023-10-20T03:35:52.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.669 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.669 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277418+0000", 2023-10-20T03:35:52.669 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3599999999999998e-06 2023-10-20T03:35:52.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.670 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.670 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.670 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277428+0000", 2023-10-20T03:35:52.670 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0034000000000001e-05 2023-10-20T03:35:52.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.671 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.671 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.671 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277435+0000", 2023-10-20T03:35:52.671 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2049999999999999e-06 2023-10-20T03:35:52.672 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.672 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.672 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.672 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277660+0000", 2023-10-20T03:35:52.672 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022565199999999999 2023-10-20T03:35:52.672 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.673 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.673 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:52.673 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277673+0000", 2023-10-20T03:35:52.673 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2330000000000001e-05 2023-10-20T03:35:52.673 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.674 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.282724+0000", 2023-10-20T03:35:52.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0050513370000000004 2023-10-20T03:35:52.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:52.675 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.282891+0000", 2023-10-20T03:35:52.675 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016650500000000001 2023-10-20T03:35:52.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.676 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.676 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.286151+0000", 2023-10-20T03:35:52.677 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003260878 2023-10-20T03:35:52.677 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.677 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.677 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.678 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:214 3.5s0 3:aed1dba0:::smithi17039300-29:2 [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:35:52.678 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:19.277483+0000", 2023-10-20T03:35:52.678 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 33.001569867999997, 2023-10-20T03:35:52.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.009248025, 2023-10-20T03:35:52.679 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.679 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:52.679 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:52.679 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:52.680 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:52.680 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 214 2023-10-20T03:35:52.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.680 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.681 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.681 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277483+0000", 2023-10-20T03:35:52.681 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.682 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.682 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277483+0000", 2023-10-20T03:35:52.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.683 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.683 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277483+0000", 2023-10-20T03:35:52.683 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.75e-07 2023-10-20T03:35:52.683 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.683 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277489+0000", 2023-10-20T03:35:52.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.57e-06 2023-10-20T03:35:52.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.685 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.685 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.685 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277497+0000", 2023-10-20T03:35:52.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4419999999999998e-06 2023-10-20T03:35:52.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.686 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277504+0000", 2023-10-20T03:35:52.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3430000000000001e-06 2023-10-20T03:35:52.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277741+0000", 2023-10-20T03:35:52.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00023721699999999999 2023-10-20T03:35:52.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.688 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:52.688 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.277751+0000", 2023-10-20T03:35:52.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0122999999999999e-05 2023-10-20T03:35:52.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.689 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283390+0000", 2023-10-20T03:35:52.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056394009999999996 2023-10-20T03:35:52.690 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:52.690 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.283539+0000", 2023-10-20T03:35:52.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014805499999999999 2023-10-20T03:35:52.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.691 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.691 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:19.286731+0000", 2023-10-20T03:35:52.692 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003192299 2023-10-20T03:35:52.692 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.692 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.692 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.693 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:316 3.5s0 3:aed1dba0:::smithi17039300-29:head [append 0~0] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:52.693 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.364850+0000", 2023-10-20T03:35:52.693 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.914203088000001, 2023-10-20T03:35:52.693 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016187792999999999, 2023-10-20T03:35:52.694 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.694 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:52.694 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:52.694 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:52.694 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 316 2023-10-20T03:35:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364850+0000", 2023-10-20T03:35:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364850+0000", 2023-10-20T03:35:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364852+0000", 2023-10-20T03:35:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0260000000000002e-06 2023-10-20T03:35:52.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.699 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.699 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.699 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364861+0000", 2023-10-20T03:35:52.699 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5899999999999997e-06 2023-10-20T03:35:52.699 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.700 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.700 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364880+0000", 2023-10-20T03:35:52.700 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9245999999999998e-05 2023-10-20T03:35:52.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364892+0000", 2023-10-20T03:35:52.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.184e-05 2023-10-20T03:35:52.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.702 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.702 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.702 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364987+0000", 2023-10-20T03:35:52.702 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4752000000000004e-05 2023-10-20T03:35:52.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.703 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.703 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2023-10-20T03:35:52.703 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364997+0000", 2023-10-20T03:35:52.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0319999999999999e-05 2023-10-20T03:35:52.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.704 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.704 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.368108+0000", 2023-10-20T03:35:52.705 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0031111200000000002 2023-10-20T03:35:52.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.705 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:52.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.368196+0000", 2023-10-20T03:35:52.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7773000000000001e-05 2023-10-20T03:35:52.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.707 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.368815+0000", 2023-10-20T03:35:52.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00061905900000000004 2023-10-20T03:35:52.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370813+0000", 2023-10-20T03:35:52.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019981780000000002 2023-10-20T03:35:52.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:52.709 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.380943+0000", 2023-10-20T03:35:52.709 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010129487 2023-10-20T03:35:52.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.709 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.710 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.381037+0000", 2023-10-20T03:35:52.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4401999999999995e-05 2023-10-20T03:35:52.710 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.710 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.711 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.711 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:317 3.5s0 3:aed1dba0:::smithi17039300-29:head [rollback 9] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:52.711 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.364924+0000", 2023-10-20T03:35:52.712 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.914128229999999, 2023-10-20T03:35:52.712 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017874489, 2023-10-20T03:35:52.712 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.712 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:52.712 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:52.713 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:52.713 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:52.713 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 317 2023-10-20T03:35:52.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.713 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364924+0000", 2023-10-20T03:35:52.714 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.715 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.715 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.715 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364924+0000", 2023-10-20T03:35:52.716 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.716 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.716 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.716 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364925+0000", 2023-10-20T03:35:52.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8599999999999998e-07 2023-10-20T03:35:52.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.717 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.717 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364929+0000", 2023-10-20T03:35:52.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7309999999999999e-06 2023-10-20T03:35:52.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.718 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.719 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364940+0000", 2023-10-20T03:35:52.719 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1236999999999999e-05 2023-10-20T03:35:52.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.719 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.720 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364947+0000", 2023-10-20T03:35:52.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0469999999999996e-06 2023-10-20T03:35:52.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365101+0000", 2023-10-20T03:35:52.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015391 2023-10-20T03:35:52.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.721 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:52.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365110+0000", 2023-10-20T03:35:52.722 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2e-06 2023-10-20T03:35:52.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.722 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.722 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.722 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.369303+0000", 2023-10-20T03:35:52.723 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041924149999999997 2023-10-20T03:35:52.723 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.723 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.723 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:52.723 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.369380+0000", 2023-10-20T03:35:52.724 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.716e-05 2023-10-20T03:35:52.724 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.724 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.724 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.725 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.369806+0000", 2023-10-20T03:35:52.725 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00042586099999999998 2023-10-20T03:35:52.725 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.725 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.725 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.726 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.372757+0000", 2023-10-20T03:35:52.726 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029512430000000001 2023-10-20T03:35:52.726 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.726 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.726 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:52.727 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.382729+0000", 2023-10-20T03:35:52.727 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099724180000000003 2023-10-20T03:35:52.727 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.727 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.727 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.728 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.382799+0000", 2023-10-20T03:35:52.728 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9680999999999999e-05 2023-10-20T03:35:52.728 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.728 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.729 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.729 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.729 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.729 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:318 3.5s0 3:aed1dba0:::smithi17039300-29:head [append 0~0] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:35:52.729 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:35:52.730 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.914068077, 2023-10-20T03:35:52.730 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018045379, 2023-10-20T03:35:52.730 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.730 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:52.730 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:52.731 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:52.731 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:52.731 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 318 2023-10-20T03:35:52.731 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.732 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.732 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.732 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.732 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:35:52.732 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.733 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.733 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.733 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.733 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:35:52.734 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.734 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.734 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.734 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.734 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:35:52.735 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8899999999999999e-07 2023-10-20T03:35:52.735 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.735 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.735 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.735 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364989+0000", 2023-10-20T03:35:52.736 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7079999999999999e-06 2023-10-20T03:35:52.736 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.736 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.736 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.736 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364998+0000", 2023-10-20T03:35:52.737 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2229999999999996e-06 2023-10-20T03:35:52.737 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.737 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.737 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.738 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365004+0000", 2023-10-20T03:35:52.738 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1999999999999999e-06 2023-10-20T03:35:52.738 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.738 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.738 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.739 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365157+0000", 2023-10-20T03:35:52.739 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015222799999999999 2023-10-20T03:35:52.739 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.739 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.739 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:52.740 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365166+0000", 2023-10-20T03:35:52.740 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6739999999999992e-06 2023-10-20T03:35:52.740 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.740 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.740 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.741 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370187+0000", 2023-10-20T03:35:52.741 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0050205279999999998 2023-10-20T03:35:52.741 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.741 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.741 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:52.742 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370256+0000", 2023-10-20T03:35:52.742 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9151000000000003e-05 2023-10-20T03:35:52.742 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.742 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.743 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.743 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370618+0000", 2023-10-20T03:35:52.743 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00036223100000000002 2023-10-20T03:35:52.743 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.744 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.744 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.744 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.375514+0000", 2023-10-20T03:35:52.744 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0048961220000000001 2023-10-20T03:35:52.744 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.745 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.745 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:52.745 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.382954+0000", 2023-10-20T03:35:52.745 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074393280000000003 2023-10-20T03:35:52.746 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.746 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.746 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.746 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.383030+0000", 2023-10-20T03:35:52.746 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6396999999999994e-05 2023-10-20T03:35:52.747 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.747 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.747 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.747 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.747 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.748 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:52.748 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:35:52.748 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.890386969000001, 2023-10-20T03:35:52.748 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028147078999999998, 2023-10-20T03:35:52.748 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.749 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:52.749 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.749 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.749 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.750 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:35:52.750 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.750 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.750 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.750 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.751 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:35:52.751 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.751 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.751 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.751 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.752 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:35:52.752 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4600000000000004e-07 2023-10-20T03:35:52.752 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.752 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.753 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.753 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388785+0000", 2023-10-20T03:35:52.753 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000118604 2023-10-20T03:35:52.753 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.753 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.754 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.754 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388827+0000", 2023-10-20T03:35:52.754 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2404000000000003e-05 2023-10-20T03:35:52.754 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.755 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.755 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.755 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388843+0000", 2023-10-20T03:35:52.755 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5894000000000001e-05 2023-10-20T03:35:52.755 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.756 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.756 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.756 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.389002+0000", 2023-10-20T03:35:52.756 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000158383 2023-10-20T03:35:52.756 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.389019+0000", 2023-10-20T03:35:52.757 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7266e-05 2023-10-20T03:35:52.758 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.758 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.758 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.758 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416738+0000", 2023-10-20T03:35:52.758 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027718779999999998 2023-10-20T03:35:52.759 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.759 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.759 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.759 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416813+0000", 2023-10-20T03:35:52.759 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5102e-05 2023-10-20T03:35:52.760 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.760 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.760 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.760 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.760 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.761 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:340 3.5s0 3:aed1dba0:::smithi17039300-29:head [create,setxattr nWX_d[W\\dXUWOgbXndrSbcTU`mgjR_]o\\nNcXpin (1403) in=40b,setxattr TSP[ddnm_WR]nh]QhSXpkpmmWeOP^b]baemT^Uqr (630) in=40b,setxattr R[QU^NWYrjfarmhlWmZlUkP\\ipWan`SnqZOX`VWh (806) in=40b,setxattr ]hV\\^[PQZO]n^abhilRfldoUhgtl\\\\oel`mgahd_ (3368) in=40b,setxattr OYc`XQ\\VjZfQ`]OsbemUo_mfVUSc^pNo\\[\\ldrhg (1087) in=40b,setxattr `pZtTmno_eqZOONfatgZ_`nnhemaaQWiteO]dc[V (175) in=40b,setxattr PdQa`feQapkrnjfOUYUeqsOt_hrsg\\noqrcV]pSh (981) in=40b,setxattr bk_QpaSsdlQfT`oa[pTQS_[ad]isfYm^tQNdhSiW (1718) in=40b,setxattr Ys`Y_UZ^PXhpgS`VS]`eVgkcmT^QW[aWscssRRSr (10033) in=40b,setxattr [jd]Q\\qemWTVcc`hiZnsrqrggetajZRa]\\_llS`h (1367) in=40b,setxattr `UajoeZh]tRSf\\[nnVat^TjN^_q`ooQ_br`itgZm (244) in=40b,setxattr Tng[WcWa_dsnRXtZNa^`fle`gQ___mar\\fgRsZVY (665) in=40b,setxattr nogmhepVZfQtnZc^cZSeaaogjklZkrff_Utg`UR^ (15856) in=40b,setxattr gfqSsQY[][rrac[gmaQrTSacNWttTTXhanmRa^fa (1998) in=40b,setxattr V`Y[ddb`_dfep[_]oUoTqQj\\UW[gkesnlgNPYtPa (1438) in=40b,setxattr YgoRVqUOj_mXPXsq^PdjaaolmQsTsTsZpaddktl[ (1400) in=40b,setxattr smn[QVPVkfVs]oTip]jtirOtU\\Pa\\Ym^poZgfZaq (1263) in=40b,setxattr TQSmSs^Z]\\rdosO_a^TOU^[p`jkQb`dXXh]s`Stq (909) in=40b,setxattr faPhpqVXmWh`VeNZo[b^[kYl^[[rYYPRrk_[ON`e (14664) in=40b,setxattr ksO_UmqqQefe[Q\\njt[W_roWeei`]Sn^TV]O^iSW (1963) in=40b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:52.761 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.397436+0000", 2023-10-20T03:35:52.761 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.881616272999999, 2023-10-20T03:35:52.761 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024477860000000001, 2023-10-20T03:35:52.762 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.762 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:52.762 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:52.762 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:52.762 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:52.763 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 340 2023-10-20T03:35:52.763 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.763 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.763 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.764 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.764 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397436+0000", 2023-10-20T03:35:52.764 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.764 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.765 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.765 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.765 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397436+0000", 2023-10-20T03:35:52.765 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.765 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.766 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.766 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.766 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397439+0000", 2023-10-20T03:35:52.766 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6929999999999999e-06 2023-10-20T03:35:52.767 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.767 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.767 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.767 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397608+0000", 2023-10-20T03:35:52.767 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016918 2023-10-20T03:35:52.768 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.768 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.768 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.768 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397637+0000", 2023-10-20T03:35:52.768 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9057e-05 2023-10-20T03:35:52.769 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.769 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.769 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.769 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397653+0000", 2023-10-20T03:35:52.769 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5430999999999998e-05 2023-10-20T03:35:52.770 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.770 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.770 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.770 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397749+0000", 2023-10-20T03:35:52.771 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6694000000000006e-05 2023-10-20T03:35:52.771 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.771 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.771 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:52.772 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397877+0000", 2023-10-20T03:35:52.772 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012762599999999999 2023-10-20T03:35:52.772 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.772 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.772 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.773 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.399009+0000", 2023-10-20T03:35:52.773 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011323640000000001 2023-10-20T03:35:52.773 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.773 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.773 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.774 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416407+0000", 2023-10-20T03:35:52.774 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017398034999999999 2023-10-20T03:35:52.774 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.774 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.774 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:52.775 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.421802+0000", 2023-10-20T03:35:52.775 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053944320000000002 2023-10-20T03:35:52.775 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.775 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.776 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.776 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.421914+0000", 2023-10-20T03:35:52.776 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000112348 2023-10-20T03:35:52.776 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.776 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.777 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.777 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.777 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.777 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:52.778 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.401616+0000", 2023-10-20T03:35:52.778 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.877436518, 2023-10-20T03:35:52.778 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015282946, 2023-10-20T03:35:52.778 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.778 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:52.779 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.779 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.779 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.779 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401616+0000", 2023-10-20T03:35:52.780 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.780 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.780 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.780 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.780 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401616+0000", 2023-10-20T03:35:52.781 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.781 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.781 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.781 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.781 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401617+0000", 2023-10-20T03:35:52.782 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.105e-06 2023-10-20T03:35:52.782 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.782 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.782 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.783 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401748+0000", 2023-10-20T03:35:52.783 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013050599999999999 2023-10-20T03:35:52.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.783 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.783 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.784 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401765+0000", 2023-10-20T03:35:52.784 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7450999999999999e-05 2023-10-20T03:35:52.784 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.784 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.785 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.785 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401774+0000", 2023-10-20T03:35:52.785 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9760000000000001e-06 2023-10-20T03:35:52.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.786 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.786 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404428+0000", 2023-10-20T03:35:52.786 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00265409 2023-10-20T03:35:52.786 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.786 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.787 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.787 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404442+0000", 2023-10-20T03:35:52.787 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3602e-05 2023-10-20T03:35:52.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.788 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.788 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.788 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416847+0000", 2023-10-20T03:35:52.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012404837 2023-10-20T03:35:52.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.789 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.789 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416899+0000", 2023-10-20T03:35:52.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2379000000000001e-05 2023-10-20T03:35:52.790 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.790 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.790 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.790 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.791 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=53, reqid=client.4570.0:335, at_version=44'20, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:52.791 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.403071+0000", 2023-10-20T03:35:52.791 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.875981518, 2023-10-20T03:35:52.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014226947, 2023-10-20T03:35:52.792 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.792 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:52.792 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.792 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.793 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403071+0000", 2023-10-20T03:35:52.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403071+0000", 2023-10-20T03:35:52.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403072+0000", 2023-10-20T03:35:52.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2299999999999999e-07 2023-10-20T03:35:52.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.796 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403107+0000", 2023-10-20T03:35:52.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4900000000000001e-05 2023-10-20T03:35:52.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.797 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.797 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403125+0000", 2023-10-20T03:35:52.797 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8386999999999999e-05 2023-10-20T03:35:52.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.798 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.798 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403138+0000", 2023-10-20T03:35:52.798 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2979999999999999e-05 2023-10-20T03:35:52.798 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.799 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.799 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.799 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413715+0000", 2023-10-20T03:35:52.799 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010576538999999999 2023-10-20T03:35:52.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.800 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.800 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.800 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413733+0000", 2023-10-20T03:35:52.800 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8357000000000001e-05 2023-10-20T03:35:52.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.801 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.801 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417237+0000", 2023-10-20T03:35:52.801 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0035037689999999999 2023-10-20T03:35:52.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.802 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.802 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.802 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417298+0000", 2023-10-20T03:35:52.803 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1092000000000004e-05 2023-10-20T03:35:52.803 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.803 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.803 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.804 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.804 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 44/31 ECSubWriteReply(tid=22, last_complete=44'12, committed=1, applied=1))", 2023-10-20T03:35:52.804 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.404745+0000", 2023-10-20T03:35:52.804 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.874308066000001, 2023-10-20T03:35:52.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089739039999999996, 2023-10-20T03:35:52.805 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.805 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:52.805 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.806 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.806 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404745+0000", 2023-10-20T03:35:52.806 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404745+0000", 2023-10-20T03:35:52.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.808 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.808 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404746+0000", 2023-10-20T03:35:52.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.156e-06 2023-10-20T03:35:52.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.809 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.809 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.809 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404753+0000", 2023-10-20T03:35:52.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7009999999999996e-06 2023-10-20T03:35:52.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404774+0000", 2023-10-20T03:35:52.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0390999999999998e-05 2023-10-20T03:35:52.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404790+0000", 2023-10-20T03:35:52.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5744e-05 2023-10-20T03:35:52.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413701+0000", 2023-10-20T03:35:52.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089117050000000007 2023-10-20T03:35:52.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413719+0000", 2023-10-20T03:35:52.814 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7207e-05 2023-10-20T03:35:52.814 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.814 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.814 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.815 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.815 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=54, reqid=client.4570.0:336, at_version=44'21, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:35:52.815 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.405096+0000", 2023-10-20T03:35:52.815 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.873956460999999, 2023-10-20T03:35:52.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012879895000000001, 2023-10-20T03:35:52.816 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.816 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:52.816 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.817 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.817 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405096+0000", 2023-10-20T03:35:52.817 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.818 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.818 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405096+0000", 2023-10-20T03:35:52.818 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.819 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.819 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.819 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405097+0000", 2023-10-20T03:35:52.819 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.035e-06 2023-10-20T03:35:52.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.820 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.820 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.820 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405121+0000", 2023-10-20T03:35:52.820 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4097e-05 2023-10-20T03:35:52.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.821 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405177+0000", 2023-10-20T03:35:52.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5220000000000003e-05 2023-10-20T03:35:52.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.822 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.822 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405192+0000", 2023-10-20T03:35:52.822 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.579e-05 2023-10-20T03:35:52.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.823 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.823 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.823 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416447+0000", 2023-10-20T03:35:52.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011254188 2023-10-20T03:35:52.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.824 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.824 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416463+0000", 2023-10-20T03:35:52.824 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6441000000000001e-05 2023-10-20T03:35:52.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.825 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.825 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417913+0000", 2023-10-20T03:35:52.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014495809999999999 2023-10-20T03:35:52.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.826 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.826 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417976+0000", 2023-10-20T03:35:52.827 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3542999999999998e-05 2023-10-20T03:35:52.827 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.827 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.827 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.828 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.828 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 44/31 ECSubWriteReply(tid=22, last_complete=44'12, committed=1, applied=1))", 2023-10-20T03:35:52.828 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.406456+0000", 2023-10-20T03:35:52.828 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.872596612999999, 2023-10-20T03:35:52.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010387031999999999, 2023-10-20T03:35:52.829 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.829 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:52.829 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406456+0000", 2023-10-20T03:35:52.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406456+0000", 2023-10-20T03:35:52.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.832 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.832 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406457+0000", 2023-10-20T03:35:52.832 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6199999999999996e-07 2023-10-20T03:35:52.832 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.833 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.833 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406462+0000", 2023-10-20T03:35:52.833 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2889999999999997e-06 2023-10-20T03:35:52.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.834 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.834 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.834 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406479+0000", 2023-10-20T03:35:52.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6963e-05 2023-10-20T03:35:52.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.835 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.835 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406488+0000", 2023-10-20T03:35:52.835 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6200000000000005e-06 2023-10-20T03:35:52.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.836 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.836 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416476+0000", 2023-10-20T03:35:52.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099880660000000003 2023-10-20T03:35:52.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.837 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.837 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416843+0000", 2023-10-20T03:35:52.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000367232 2023-10-20T03:35:52.837 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.838 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.838 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.838 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 44/31 ECSubWriteReply(tid=22, last_complete=44'12, committed=1, applied=1))", 2023-10-20T03:35:52.839 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.406529+0000", 2023-10-20T03:35:52.839 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.872523358999999, 2023-10-20T03:35:52.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010245637, 2023-10-20T03:35:52.839 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.840 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:52.840 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.840 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406529+0000", 2023-10-20T03:35:52.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406529+0000", 2023-10-20T03:35:52.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.843 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.843 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406530+0000", 2023-10-20T03:35:52.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9599999999999999e-07 2023-10-20T03:35:52.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.844 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406541+0000", 2023-10-20T03:35:52.844 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1208e-05 2023-10-20T03:35:52.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.845 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.845 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406557+0000", 2023-10-20T03:35:52.845 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5563999999999999e-05 2023-10-20T03:35:52.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406566+0000", 2023-10-20T03:35:52.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0769999999999997e-06 2023-10-20T03:35:52.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416751+0000", 2023-10-20T03:35:52.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010184794 2023-10-20T03:35:52.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416775+0000", 2023-10-20T03:35:52.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4298e-05 2023-10-20T03:35:52.849 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.849 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.849 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.850 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.850 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:35:52.850 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.393394+0000", 2023-10-20T03:35:52.850 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.885658170999999, 2023-10-20T03:35:52.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039046538999999998, 2023-10-20T03:35:52.851 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.851 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:52.851 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393394+0000", 2023-10-20T03:35:52.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.853 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393394+0000", 2023-10-20T03:35:52.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.854 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.854 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393395+0000", 2023-10-20T03:35:52.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.16e-07 2023-10-20T03:35:52.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.855 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.855 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394217+0000", 2023-10-20T03:35:52.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00082179599999999996 2023-10-20T03:35:52.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.856 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.856 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394258+0000", 2023-10-20T03:35:52.856 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0546000000000001e-05 2023-10-20T03:35:52.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394283+0000", 2023-10-20T03:35:52.857 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5707e-05 2023-10-20T03:35:52.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394593+0000", 2023-10-20T03:35:52.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031000800000000002 2023-10-20T03:35:52.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.859 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.859 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394630+0000", 2023-10-20T03:35:52.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6885999999999999e-05 2023-10-20T03:35:52.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432353+0000", 2023-10-20T03:35:52.861 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037723055999999998 2023-10-20T03:35:52.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.861 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.861 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432441+0000", 2023-10-20T03:35:52.862 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7724000000000002e-05 2023-10-20T03:35:52.862 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.862 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.862 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.863 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:392 3.5s0 3:aed1dba0:::smithi17039300-29:head [stat,delete] snapc f=[f,d,b,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:35:52.863 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.382023+0000", 2023-10-20T03:35:52.863 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.897029709999998, 2023-10-20T03:35:52.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019348521, 2023-10-20T03:35:52.864 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.864 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:52.864 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:52.865 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:52.865 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:52.865 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 392 2023-10-20T03:35:52.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.865 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382023+0000", 2023-10-20T03:35:52.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382023+0000", 2023-10-20T03:35:52.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382026+0000", 2023-10-20T03:35:52.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3629999999999998e-06 2023-10-20T03:35:52.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382040+0000", 2023-10-20T03:35:52.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3913e-05 2023-10-20T03:35:52.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382075+0000", 2023-10-20T03:35:52.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4678999999999998e-05 2023-10-20T03:35:52.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382093+0000", 2023-10-20T03:35:52.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7779999999999999e-05 2023-10-20T03:35:52.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.872 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382244+0000", 2023-10-20T03:35:52.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015178300000000001 2023-10-20T03:35:52.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2023-10-20T03:35:52.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382257+0000", 2023-10-20T03:35:52.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.288e-05 2023-10-20T03:35:52.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.387421+0000", 2023-10-20T03:35:52.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0051631560000000003 2023-10-20T03:35:52.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:52.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.387619+0000", 2023-10-20T03:35:52.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000198712 2023-10-20T03:35:52.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.389154+0000", 2023-10-20T03:35:52.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001535193 2023-10-20T03:35:52.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.393045+0000", 2023-10-20T03:35:52.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0038909639999999998 2023-10-20T03:35:52.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:52.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.401188+0000", 2023-10-20T03:35:52.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081430219999999998 2023-10-20T03:35:52.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.880 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.880 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.401371+0000", 2023-10-20T03:35:52.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018307600000000001 2023-10-20T03:35:52.880 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.880 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.881 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.881 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s1 70/56 ECSubWrite(tid=55, reqid=client.4570.0:556, at_version=70'40, trim_to=0'0, roll_forward_to=60'38))", 2023-10-20T03:35:52.881 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:51.966548+0000", 2023-10-20T03:35:52.882 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.31250489199999998, 2023-10-20T03:35:52.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092291170000000002, 2023-10-20T03:35:52.882 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.882 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:52.883 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.966548+0000", 2023-10-20T03:35:52.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.966548+0000", 2023-10-20T03:35:52.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.966548+0000", 2023-10-20T03:35:52.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2099999999999998e-07 2023-10-20T03:35:52.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.886 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.886 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.967078+0000", 2023-10-20T03:35:52.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053029399999999995 2023-10-20T03:35:52.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.967098+0000", 2023-10-20T03:35:52.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9848999999999999e-05 2023-10-20T03:35:52.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.967112+0000", 2023-10-20T03:35:52.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3448000000000001e-05 2023-10-20T03:35:52.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.967247+0000", 2023-10-20T03:35:52.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000134974 2023-10-20T03:35:52.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.890 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.967264+0000", 2023-10-20T03:35:52.891 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7751999999999999e-05 2023-10-20T03:35:52.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.891 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.892 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.975739+0000", 2023-10-20T03:35:52.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0084742600000000008 2023-10-20T03:35:52.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.975777+0000", 2023-10-20T03:35:52.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8219000000000001e-05 2023-10-20T03:35:52.893 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.893 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.894 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.894 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s1 70/56 ECSubWrite(tid=57, reqid=client.4570.0:558, at_version=70'42, trim_to=0'0, roll_forward_to=70'40))", 2023-10-20T03:35:52.894 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:51.998966+0000", 2023-10-20T03:35:52.895 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.280086373, 2023-10-20T03:35:52.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0088028599999999992, 2023-10-20T03:35:52.895 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:52.895 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:52.896 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:52.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:52.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998966+0000", 2023-10-20T03:35:52.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.897 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.897 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:52.897 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998966+0000", 2023-10-20T03:35:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998969+0000", 2023-10-20T03:35:52.899 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0029999999999999e-06 2023-10-20T03:35:52.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.899 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:52.900 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.999245+0000", 2023-10-20T03:35:52.900 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00027532199999999999 2023-10-20T03:35:52.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.900 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:52.901 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.999279+0000", 2023-10-20T03:35:52.901 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4570000000000003e-05 2023-10-20T03:35:52.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.901 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.902 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:52.902 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.999303+0000", 2023-10-20T03:35:52.902 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3909000000000001e-05 2023-10-20T03:35:52.902 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.902 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.903 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:52.903 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.999404+0000", 2023-10-20T03:35:52.903 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000100754 2023-10-20T03:35:52.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.904 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.904 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:52.904 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.999417+0000", 2023-10-20T03:35:52.904 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2806000000000001e-05 2023-10-20T03:35:52.904 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.905 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.905 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:52.905 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.007646+0000", 2023-10-20T03:35:52.905 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0082295179999999999 2023-10-20T03:35:52.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:52.906 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:52.906 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:52.906 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.007769+0000", 2023-10-20T03:35:52.906 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012297799999999999 2023-10-20T03:35:52.907 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.907 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.907 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.907 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:52.907 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:52.908 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:52.908 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-10-20T03:35:52.911 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:52.906+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:52.953 INFO:tasks.rados.rados.0.smithi170.stdout:238: done (2 left) 2023-10-20T03:35:52.953 INFO:tasks.rados.rados.0.smithi170.stdout:239: rmattr oid 50 current snap is 18 2023-10-20T03:35:52.954 INFO:tasks.rados.rados.0.smithi170.stdout:239: done (2 left) 2023-10-20T03:35:52.954 INFO:tasks.rados.rados.0.smithi170.stdout:240: setattr oid 22 current snap is 18 2023-10-20T03:35:52.954 INFO:tasks.rados.rados.0.smithi170.stdout:234: finishing write tid 2 to smithi17039300-11 2023-10-20T03:35:52.954 INFO:tasks.rados.rados.0.smithi170.stdout:234: finishing write tid 3 to smithi17039300-11 2023-10-20T03:35:52.954 INFO:tasks.rados.rados.0.smithi170.stdout:234: finishing write tid 4 to smithi17039300-11 2023-10-20T03:35:52.955 INFO:tasks.rados.rados.0.smithi170.stdout:234: finishing write tid 5 to smithi17039300-11 2023-10-20T03:35:52.955 INFO:tasks.rados.rados.0.smithi170.stdout:234: finishing write tid 6 to smithi17039300-11 2023-10-20T03:35:52.955 INFO:tasks.rados.rados.0.smithi170.stdout:234: oid 11 updating version 0 to 36 2023-10-20T03:35:52.955 INFO:tasks.rados.rados.0.smithi170.stdout:234: oid 11 updating version 36 to 37 2023-10-20T03:35:52.955 INFO:tasks.rados.rados.0.smithi170.stdout:234: oid 11 updating version 37 to 40 2023-10-20T03:35:52.956 INFO:tasks.rados.rados.0.smithi170.stdout:234: oid 11 version 40 is already newer than 39 2023-10-20T03:35:52.956 INFO:tasks.rados.rados.0.smithi170.stdout:234: oid 11 version 40 is already newer than 38 2023-10-20T03:35:52.956 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 11 v 40 (ObjNum 108 snap 18 seq_num 108) dirty exists 2023-10-20T03:35:52.956 INFO:tasks.rados.rados.0.smithi170.stdout:234: left oid 11 (ObjNum 108 snap 18 seq_num 108) 2023-10-20T03:35:52.956 INFO:tasks.rados.rados.0.smithi170.stdout:236: finishing write tid 1 to smithi17039300-24 2023-10-20T03:35:52.957 INFO:tasks.rados.rados.0.smithi170.stdout:236: finishing write tid 2 to smithi17039300-24 2023-10-20T03:35:52.957 INFO:tasks.rados.rados.0.smithi170.stdout:236: finishing write tid 3 to smithi17039300-24 2023-10-20T03:35:52.957 INFO:tasks.rados.rados.0.smithi170.stdout:236: finishing write tid 4 to smithi17039300-24 2023-10-20T03:35:52.957 INFO:tasks.rados.rados.0.smithi170.stdout:236: finishing write tid 5 to smithi17039300-24 2023-10-20T03:35:52.958 INFO:tasks.rados.rados.0.smithi170.stdout:236: oid 24 updating version 0 to 39 2023-10-20T03:35:52.958 INFO:tasks.rados.rados.0.smithi170.stdout:236: oid 24 version 39 is already newer than 36 2023-10-20T03:35:52.958 INFO:tasks.rados.rados.0.smithi170.stdout:236: oid 24 version 39 is already newer than 38 2023-10-20T03:35:52.958 INFO:tasks.rados.rados.0.smithi170.stdout:236: oid 24 version 39 is already newer than 37 2023-10-20T03:35:52.959 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 24 v 39 (ObjNum 109 snap 18 seq_num 109) dirty exists 2023-10-20T03:35:52.959 INFO:tasks.rados.rados.0.smithi170.stdout:236: left oid 24 (ObjNum 109 snap 18 seq_num 109) 2023-10-20T03:35:52.959 INFO:tasks.rados.rados.0.smithi170.stdout:234: done (2 left) 2023-10-20T03:35:52.960 INFO:tasks.rados.rados.0.smithi170.stdout:236: done (1 left) 2023-10-20T03:35:52.960 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 50 current snap is 18 2023-10-20T03:35:52.960 INFO:tasks.rados.rados.0.smithi170.stdout:241: seq_num 111 ranges {2392064=704512,3096576=655360,3751936=507904} 2023-10-20T03:35:52.978 INFO:tasks.rados.rados.0.smithi170.stdout:241: writing smithi17039300-50 from 2392064 to 3096576 tid 1 2023-10-20T03:35:52.984 INFO:tasks.rados.rados.0.smithi170.stdout:241: writing smithi17039300-50 from 3096576 to 3751936 tid 2 2023-10-20T03:35:52.988 INFO:tasks.rados.rados.0.smithi170.stdout:241: writing smithi17039300-50 from 3751936 to 4259840 tid 3 2023-10-20T03:35:52.988 INFO:tasks.rados.rados.0.smithi170.stdout:242: read oid 20 snap -1 2023-10-20T03:35:52.988 INFO:tasks.rados.rados.0.smithi170.stdout:242: expect (ObjNum 47 snap 0 seq_num 47) 2023-10-20T03:35:52.989 INFO:tasks.rados.rados.0.smithi170.stdout:243: read oid 7 snap -1 2023-10-20T03:35:52.989 INFO:tasks.rados.rados.0.smithi170.stdout:243: expect (ObjNum 66 snap 4 seq_num 66) 2023-10-20T03:35:52.989 INFO:tasks.rados.rados.0.smithi170.stdout:244: read oid 10 snap -1 2023-10-20T03:35:52.989 INFO:tasks.rados.rados.0.smithi170.stdout:244: expect (ObjNum 69 snap 4 seq_num 69) 2023-10-20T03:35:52.990 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 40 current snap is 18 2023-10-20T03:35:52.990 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 26 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-10-20T03:35:52.991 INFO:tasks.rados.rados.0.smithi170.stdout:245: seq_num 112 ranges {671744=720896,1392640=802816,2195456=360448} 2023-10-20T03:35:53.000 INFO:tasks.rados.rados.0.smithi170.stdout:245: writing smithi17039300-40 from 671744 to 1392640 tid 1 2023-10-20T03:35:53.007 INFO:tasks.rados.rados.0.smithi170.stdout:245: writing smithi17039300-40 from 1392640 to 2195456 tid 2 2023-10-20T03:35:53.011 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:53.010+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:53.012 INFO:tasks.rados.rados.0.smithi170.stdout:245: writing smithi17039300-40 from 2195456 to 2555904 tid 3 2023-10-20T03:35:53.013 INFO:tasks.rados.rados.0.smithi170.stdout:240: done (5 left) 2023-10-20T03:35:53.013 INFO:tasks.rados.rados.0.smithi170.stdout:246: read oid 19 snap -1 2023-10-20T03:35:53.013 INFO:tasks.rados.rados.0.smithi170.stdout:246: expect (ObjNum 98 snap 13 seq_num 98) 2023-10-20T03:35:53.014 INFO:tasks.rados.rados.0.smithi170.stdout:247: setattr oid 27 current snap is 18 2023-10-20T03:35:53.014 INFO:tasks.rados.rados.0.smithi170.stdout:243: expect (ObjNum 66 snap 4 seq_num 66) 2023-10-20T03:35:53.027 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-10-20T03:35:53.056 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:53.056 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:35:53.056 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:53.077 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-10-20T03:35:53.082 INFO:tasks.rados.rados.0.smithi170.stdout:244: expect (ObjNum 69 snap 4 seq_num 69) 2023-10-20T03:35:53.085 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:53.085 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.119 INFO:tasks.rados.rados.0.smithi170.stdout:242: expect (ObjNum 47 snap 0 seq_num 47) 2023-10-20T03:35:53.119 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:53.120 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.120 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:53.117+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:53.140 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:53.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.142 INFO:tasks.rados.rados.0.smithi170.stdout:241: finishing write tid 1 to smithi17039300-50 2023-10-20T03:35:53.142 INFO:tasks.rados.rados.0.smithi170.stdout:246: expect (ObjNum 98 snap 13 seq_num 98) 2023-10-20T03:35:53.165 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:53.166 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.185 INFO:tasks.rados.rados.0.smithi170.stdout:241: finishing write tid 2 to smithi17039300-50 2023-10-20T03:35:53.185 INFO:tasks.rados.rados.0.smithi170.stdout:241: finishing write tid 3 to smithi17039300-50 2023-10-20T03:35:53.185 INFO:tasks.rados.rados.0.smithi170.stdout:241: finishing write tid 4 to smithi17039300-50 2023-10-20T03:35:53.185 INFO:tasks.rados.rados.0.smithi170.stdout:241: finishing write tid 5 to smithi17039300-50 2023-10-20T03:35:53.186 INFO:tasks.rados.rados.0.smithi170.stdout:241: oid 50 updating version 0 to 48 2023-10-20T03:35:53.186 INFO:tasks.rados.rados.0.smithi170.stdout:241: oid 50 version 48 is already newer than 46 2023-10-20T03:35:53.186 INFO:tasks.rados.rados.0.smithi170.stdout:241: oid 50 version 48 is already newer than 47 2023-10-20T03:35:53.186 INFO:tasks.rados.rados.0.smithi170.stdout:241: oid 50 updating version 48 to 49 2023-10-20T03:35:53.186 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 50 v 49 (ObjNum 111 snap 18 seq_num 111) dirty exists 2023-10-20T03:35:53.187 INFO:tasks.rados.rados.0.smithi170.stdout:241: left oid 50 (ObjNum 111 snap 18 seq_num 111) 2023-10-20T03:35:53.187 INFO:tasks.rados.rados.0.smithi170.stdout:245: finishing write tid 1 to smithi17039300-40 2023-10-20T03:35:53.187 INFO:tasks.rados.rados.0.smithi170.stdout:245: finishing write tid 2 to smithi17039300-40 2023-10-20T03:35:53.187 INFO:tasks.rados.rados.0.smithi170.stdout:245: finishing write tid 3 to smithi17039300-40 2023-10-20T03:35:53.187 INFO:tasks.rados.rados.0.smithi170.stdout:245: finishing write tid 4 to smithi17039300-40 2023-10-20T03:35:53.188 INFO:tasks.rados.rados.0.smithi170.stdout:245: finishing write tid 5 to smithi17039300-40 2023-10-20T03:35:53.188 INFO:tasks.rados.rados.0.smithi170.stdout:245: oid 40 updating version 0 to 27 2023-10-20T03:35:53.188 INFO:tasks.rados.rados.0.smithi170.stdout:245: oid 40 version 27 is already newer than 25 2023-10-20T03:35:53.188 INFO:tasks.rados.rados.0.smithi170.stdout:245: oid 40 version 27 is already newer than 26 2023-10-20T03:35:53.188 INFO:tasks.rados.rados.0.smithi170.stdout:245: oid 40 version 27 is already newer than 24 2023-10-20T03:35:53.189 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 27 (ObjNum 112 snap 18 seq_num 112) dirty exists 2023-10-20T03:35:53.189 INFO:tasks.rados.rados.0.smithi170.stdout:245: left oid 40 (ObjNum 112 snap 18 seq_num 112) 2023-10-20T03:35:53.189 INFO:tasks.rados.rados.0.smithi170.stdout:241: done (6 left) 2023-10-20T03:35:53.189 INFO:tasks.rados.rados.0.smithi170.stdout:242: done (5 left) 2023-10-20T03:35:53.190 INFO:tasks.rados.rados.0.smithi170.stdout:243: done (4 left) 2023-10-20T03:35:53.190 INFO:tasks.rados.rados.0.smithi170.stdout:244: done (3 left) 2023-10-20T03:35:53.190 INFO:tasks.rados.rados.0.smithi170.stdout:245: done (2 left) 2023-10-20T03:35:53.190 INFO:tasks.rados.rados.0.smithi170.stdout:246: done (1 left) 2023-10-20T03:35:53.190 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 20 current snap is 18 2023-10-20T03:35:53.191 INFO:tasks.rados.rados.0.smithi170.stdout:248: seq_num 113 ranges {802816=737280,1540096=557056,2097152=606208} 2023-10-20T03:35:53.200 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:53.200 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.208 INFO:tasks.rados.rados.0.smithi170.stdout:248: writing smithi17039300-20 from 802816 to 1540096 tid 1 2023-10-20T03:35:53.212 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:53.208+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:53.218 INFO:tasks.rados.rados.0.smithi170.stdout:248: writing smithi17039300-20 from 1540096 to 2097152 tid 2 2023-10-20T03:35:53.222 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:53.222 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:35:53.222 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:35:53.223 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:35:53.223 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:53.224 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.225 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:53.230 INFO:tasks.rados.rados.0.smithi170.stdout:248: writing smithi17039300-20 from 2097152 to 2703360 tid 3 2023-10-20T03:35:53.230 INFO:tasks.rados.rados.0.smithi170.stdout:249: snap_create 2023-10-20T03:35:53.230 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 27 v 28 (ObjNum 129 snap 0 seq_num 3160926096) dirty exists 2023-10-20T03:35:53.239 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-10-20T03:35:53.241 INFO:tasks.rados.rados.0.smithi170.stdout:248: finishing write tid 1 to smithi17039300-20 2023-10-20T03:35:53.253 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:53.254 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.257 INFO:tasks.rados.rados.0.smithi170.stdout:248: finishing write tid 2 to smithi17039300-20 2023-10-20T03:35:53.261 INFO:tasks.rados.rados.0.smithi170.stdout:248: finishing write tid 3 to smithi17039300-20 2023-10-20T03:35:53.275 INFO:tasks.rados.rados.0.smithi170.stdout:248: finishing write tid 4 to smithi17039300-20 2023-10-20T03:35:53.279 INFO:tasks.rados.rados.0.smithi170.stdout:248: finishing write tid 5 to smithi17039300-20 2023-10-20T03:35:53.279 INFO:tasks.rados.rados.0.smithi170.stdout:248: oid 20 updating version 0 to 28 2023-10-20T03:35:53.279 INFO:tasks.rados.rados.0.smithi170.stdout:248: oid 20 version 28 is already newer than 27 2023-10-20T03:35:53.279 INFO:tasks.rados.rados.0.smithi170.stdout:248: oid 20 updating version 28 to 29 2023-10-20T03:35:53.280 INFO:tasks.rados.rados.0.smithi170.stdout:248: oid 20 updating version 29 to 30 2023-10-20T03:35:53.280 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 20 v 30 (ObjNum 113 snap 18 seq_num 113) dirty exists 2023-10-20T03:35:53.280 INFO:tasks.rados.rados.0.smithi170.stdout:248: left oid 20 (ObjNum 113 snap 18 seq_num 113) 2023-10-20T03:35:53.292 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:53.292 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.313 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:53.312+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:53.327 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:53.327 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.360 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:35:53.361 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-10-20T03:35:53.362 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:53.363 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.373 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:35:53.373 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:35:53.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:35:53.374 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:35:53.374 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.374 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:35:53.375 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:35:53.375 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 28.014445640000002, 2023-10-20T03:35:53.375 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010383547999999999, 2023-10-20T03:35:53.375 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.375 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:53.376 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.376 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.376 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.376 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:35:53.377 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.377 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:35:53.378 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356840+0000", 2023-10-20T03:35:53.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1999999999999999e-07 2023-10-20T03:35:53.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.379 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.380 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.380 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357506+0000", 2023-10-20T03:35:53.380 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00066620500000000001 2023-10-20T03:35:53.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.380 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.381 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357522+0000", 2023-10-20T03:35:53.381 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5964000000000002e-05 2023-10-20T03:35:53.381 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.381 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.382 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.382 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357535+0000", 2023-10-20T03:35:53.382 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3046e-05 2023-10-20T03:35:53.382 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.383 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.383 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.383 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357625+0000", 2023-10-20T03:35:53.383 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9691999999999994e-05 2023-10-20T03:35:53.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.384 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.384 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:53.384 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357636+0000", 2023-10-20T03:35:53.384 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1202e-05 2023-10-20T03:35:53.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.385 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.385 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:53.385 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.367176+0000", 2023-10-20T03:35:53.385 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0095396060000000008 2023-10-20T03:35:53.385 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.386 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.386 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.386 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.367223+0000", 2023-10-20T03:35:53.386 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7113000000000001e-05 2023-10-20T03:35:53.386 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.387 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.387 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.387 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=85, reqid=client.4570.0:312, at_version=43'25, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:35:53.388 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:35:53.388 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 27.996933242000001, 2023-10-20T03:35:53.388 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0052771249999999997, 2023-10-20T03:35:53.388 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.388 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:53.388 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.389 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.389 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.389 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:35:53.389 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.390 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.390 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.390 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:35:53.391 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.391 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.391 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374353+0000", 2023-10-20T03:35:53.392 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6199999999999999e-06 2023-10-20T03:35:53.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.392 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.392 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.392 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374496+0000", 2023-10-20T03:35:53.393 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014282800000000001 2023-10-20T03:35:53.393 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.393 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.393 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.393 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374515+0000", 2023-10-20T03:35:53.393 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8281e-05 2023-10-20T03:35:53.394 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.394 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.394 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.394 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374527+0000", 2023-10-20T03:35:53.395 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2835e-05 2023-10-20T03:35:53.395 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.395 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.395 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.395 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374607+0000", 2023-10-20T03:35:53.396 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9283999999999995e-05 2023-10-20T03:35:53.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.396 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.396 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:53.396 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374617+0000", 2023-10-20T03:35:53.397 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0394000000000001e-05 2023-10-20T03:35:53.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.397 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.397 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:53.397 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.379594+0000", 2023-10-20T03:35:53.398 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049767650000000002 2023-10-20T03:35:53.398 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.398 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.398 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.399 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.379629+0000", 2023-10-20T03:35:53.399 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5117999999999998e-05 2023-10-20T03:35:53.399 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.399 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.399 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.400 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.400 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.400 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=88, reqid=client.4570.0:313, at_version=43'26, trim_to=0'0, roll_forward_to=43'24))", 2023-10-20T03:35:53.400 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:35:53.401 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 27.983742939999999, 2023-10-20T03:35:53.401 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049208230000000004, 2023-10-20T03:35:53.401 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.401 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:53.401 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.402 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.402 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.402 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:35:53.402 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.403 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.403 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.403 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.403 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:35:53.403 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.404 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.404 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.404 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.404 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387544+0000", 2023-10-20T03:35:53.404 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6109999999999999e-06 2023-10-20T03:35:53.405 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.405 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.405 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.405 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387771+0000", 2023-10-20T03:35:53.406 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022766 2023-10-20T03:35:53.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.406 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.406 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.406 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387788+0000", 2023-10-20T03:35:53.407 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7079999999999999e-05 2023-10-20T03:35:53.407 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.407 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.407 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.407 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387799+0000", 2023-10-20T03:35:53.408 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0608000000000001e-05 2023-10-20T03:35:53.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.408 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.408 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.408 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387881+0000", 2023-10-20T03:35:53.409 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2060999999999999e-05 2023-10-20T03:35:53.409 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.409 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.409 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:53.409 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387892+0000", 2023-10-20T03:35:53.410 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0912e-05 2023-10-20T03:35:53.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.410 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.410 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:53.411 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.392420+0000", 2023-10-20T03:35:53.411 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0045283320000000004 2023-10-20T03:35:53.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.411 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.412 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.412 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.392463+0000", 2023-10-20T03:35:53.412 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2558999999999997e-05 2023-10-20T03:35:53.412 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.412 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.413 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.413 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.413 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.413 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:333 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~606208] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:53.414 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:35:53.414 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.985355168000002, 2023-10-20T03:35:53.414 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036441440999999998, 2023-10-20T03:35:53.414 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.415 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:53.415 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:53.415 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:53.415 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:53.415 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 333 2023-10-20T03:35:53.416 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.416 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.416 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.416 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.417 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:35:53.417 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.417 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.417 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.417 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.418 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:35:53.418 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.418 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.418 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.419 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.419 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385932+0000", 2023-10-20T03:35:53.419 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9929999999999998e-06 2023-10-20T03:35:53.419 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.419 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.420 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.420 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386889+0000", 2023-10-20T03:35:53.420 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000957437 2023-10-20T03:35:53.420 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.420 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.421 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.421 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386904+0000", 2023-10-20T03:35:53.421 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4669e-05 2023-10-20T03:35:53.421 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.422 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.422 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.422 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386910+0000", 2023-10-20T03:35:53.422 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3040000000000001e-06 2023-10-20T03:35:53.422 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.423 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.423 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.423 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386972+0000", 2023-10-20T03:35:53.423 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2003999999999998e-05 2023-10-20T03:35:53.423 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.424 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.424 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:53.424 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387043+0000", 2023-10-20T03:35:53.424 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0591000000000002e-05 2023-10-20T03:35:53.425 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.425 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.425 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:53.425 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388080+0000", 2023-10-20T03:35:53.425 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010373190000000001 2023-10-20T03:35:53.426 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.426 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.426 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:53.426 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393723+0000", 2023-10-20T03:35:53.427 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056426899999999997 2023-10-20T03:35:53.427 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.427 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.427 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:53.427 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422308+0000", 2023-10-20T03:35:53.428 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028584822999999999 2023-10-20T03:35:53.428 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.428 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.428 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.428 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422371+0000", 2023-10-20T03:35:53.429 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3610999999999998e-05 2023-10-20T03:35:53.429 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.429 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.429 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.429 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.430 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.430 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:334 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~720896] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:53.430 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:35:53.430 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.978220769, 2023-10-20T03:35:53.430 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029571147999999998, 2023-10-20T03:35:53.431 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.431 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:53.431 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:53.431 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:53.431 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:53.432 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 334 2023-10-20T03:35:53.432 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.432 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.432 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.433 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.433 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:35:53.433 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.433 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.433 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.434 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.434 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:35:53.434 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.434 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.435 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.435 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.435 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393066+0000", 2023-10-20T03:35:53.435 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8819999999999999e-06 2023-10-20T03:35:53.435 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.436 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.436 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.436 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393827+0000", 2023-10-20T03:35:53.436 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000760844 2023-10-20T03:35:53.436 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.437 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.437 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.437 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393842+0000", 2023-10-20T03:35:53.437 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5432999999999999e-05 2023-10-20T03:35:53.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.438 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.438 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.438 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393850+0000", 2023-10-20T03:35:53.438 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0199999999999994e-06 2023-10-20T03:35:53.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.439 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.439 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.439 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393915+0000", 2023-10-20T03:35:53.439 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4123999999999998e-05 2023-10-20T03:35:53.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.440 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.440 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:53.440 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394021+0000", 2023-10-20T03:35:53.441 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010668299999999999 2023-10-20T03:35:53.441 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.441 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.441 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:53.441 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394832+0000", 2023-10-20T03:35:53.442 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081066099999999996 2023-10-20T03:35:53.442 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.442 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.442 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:53.443 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405535+0000", 2023-10-20T03:35:53.443 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010703351 2023-10-20T03:35:53.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.443 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.444 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:53.444 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422553+0000", 2023-10-20T03:35:53.444 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017018021000000001 2023-10-20T03:35:53.444 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.444 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.445 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.445 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422635+0000", 2023-10-20T03:35:53.445 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2128999999999999e-05 2023-10-20T03:35:53.445 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.445 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.446 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.446 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.446 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:335 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~49152] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:53.446 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:35:53.447 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.977413793, 2023-10-20T03:35:53.447 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029832813999999999, 2023-10-20T03:35:53.447 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.448 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:53.448 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:53.448 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:53.448 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:53.448 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 335 2023-10-20T03:35:53.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.449 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.449 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.449 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.449 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:35:53.449 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.450 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.450 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.450 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.450 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:35:53.451 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.451 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.451 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.451 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.451 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393872+0000", 2023-10-20T03:35:53.452 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6000000000000003e-07 2023-10-20T03:35:53.452 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.452 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.452 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.453 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393919+0000", 2023-10-20T03:35:53.453 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6576000000000003e-05 2023-10-20T03:35:53.453 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.453 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.453 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.454 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393928+0000", 2023-10-20T03:35:53.454 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1220000000000001e-06 2023-10-20T03:35:53.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.454 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.455 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.455 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393937+0000", 2023-10-20T03:35:53.455 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1740000000000006e-06 2023-10-20T03:35:53.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.455 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.456 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.456 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401885+0000", 2023-10-20T03:35:53.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079486149999999992 2023-10-20T03:35:53.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:53.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.402012+0000", 2023-10-20T03:35:53.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012668 2023-10-20T03:35:53.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.458 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:53.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403063+0000", 2023-10-20T03:35:53.458 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010508410000000001 2023-10-20T03:35:53.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.459 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.459 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:53.459 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405538+0000", 2023-10-20T03:35:53.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024754519999999999 2023-10-20T03:35:53.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.460 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:53.460 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.423630+0000", 2023-10-20T03:35:53.461 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018091716000000001 2023-10-20T03:35:53.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.461 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.461 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.461 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.423704+0000", 2023-10-20T03:35:53.462 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3878e-05 2023-10-20T03:35:53.462 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.462 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.462 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.463 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:336 3.ds0 3:b8ff490e:::smithi17039300-34:head [setxattr _header (56) in=7b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:53.463 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:35:53.464 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.977334418000002, 2023-10-20T03:35:53.464 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030244758, 2023-10-20T03:35:53.464 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.464 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:53.464 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:53.465 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:53.465 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:53.465 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 336 2023-10-20T03:35:53.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.465 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.466 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.466 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:35:53.466 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.467 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:35:53.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.469 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:35:53.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8400000000000004e-07 2023-10-20T03:35:53.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393955+0000", 2023-10-20T03:35:53.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8210000000000003e-06 2023-10-20T03:35:53.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393963+0000", 2023-10-20T03:35:53.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8359999999999992e-06 2023-10-20T03:35:53.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393968+0000", 2023-10-20T03:35:53.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3689999999999998e-06 2023-10-20T03:35:53.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404110+0000", 2023-10-20T03:35:53.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010141582 2023-10-20T03:35:53.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:53.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404246+0000", 2023-10-20T03:35:53.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013631799999999999 2023-10-20T03:35:53.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:53.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405009+0000", 2023-10-20T03:35:53.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00076302800000000001 2023-10-20T03:35:53.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:53.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406198+0000", 2023-10-20T03:35:53.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011889610000000001 2023-10-20T03:35:53.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:53.478 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424049+0000", 2023-10-20T03:35:53.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017850957000000001 2023-10-20T03:35:53.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.479 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.479 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424195+0000", 2023-10-20T03:35:53.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014630200000000001 2023-10-20T03:35:53.479 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.480 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.480 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.480 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:337 3.ds0 3:b8ff490e:::smithi17039300-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:53.481 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:35:53.481 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.977307633999999, 2023-10-20T03:35:53.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.032645448000000001, 2023-10-20T03:35:53.481 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.482 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:53.482 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:53.482 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:53.482 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:53.482 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 337 2023-10-20T03:35:53.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.483 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:35:53.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.485 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:35:53.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.485 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393978+0000", 2023-10-20T03:35:53.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0800000000000005e-07 2023-10-20T03:35:53.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393981+0000", 2023-10-20T03:35:53.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7199999999999998e-06 2023-10-20T03:35:53.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393988+0000", 2023-10-20T03:35:53.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5939999999999996e-06 2023-10-20T03:35:53.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393993+0000", 2023-10-20T03:35:53.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0200000000000002e-06 2023-10-20T03:35:53.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405399+0000", 2023-10-20T03:35:53.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011406229 2023-10-20T03:35:53.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:53.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405522+0000", 2023-10-20T03:35:53.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012226199999999999 2023-10-20T03:35:53.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424833+0000", 2023-10-20T03:35:53.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019311524 2023-10-20T03:35:53.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:53.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424976+0000", 2023-10-20T03:35:53.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014297600000000001 2023-10-20T03:35:53.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.426623+0000", 2023-10-20T03:35:53.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0016466149999999999 2023-10-20T03:35:53.495 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.495 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.495 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.496 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:384 3.ds0 3:ba203010:::smithi17039300-4:head [copy-from ver 21] snapc f=[f,d,b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:35:53.496 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:35:53.496 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 23.000524351999999, 2023-10-20T03:35:53.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.077402023, 2023-10-20T03:35:53.496 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.497 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:53.497 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:53.497 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:53.497 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:53.497 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 384 2023-10-20T03:35:53.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.498 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:35:53.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:35:53.501 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370763+0000", 2023-10-20T03:35:53.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4569999999999999e-06 2023-10-20T03:35:53.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370776+0000", 2023-10-20T03:35:53.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2483000000000001e-05 2023-10-20T03:35:53.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370809+0000", 2023-10-20T03:35:53.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2916000000000001e-05 2023-10-20T03:35:53.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.505 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.505 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370826+0000", 2023-10-20T03:35:53.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7183000000000001e-05 2023-10-20T03:35:53.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371075+0000", 2023-10-20T03:35:53.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000249387 2023-10-20T03:35:53.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:53.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371389+0000", 2023-10-20T03:35:53.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031429899999999998 2023-10-20T03:35:53.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.508 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:53.509 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392222+0000", 2023-10-20T03:35:53.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020832532000000001 2023-10-20T03:35:53.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:53.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.447199+0000", 2023-10-20T03:35:53.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054976917 2023-10-20T03:35:53.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:35:53.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.448009+0000", 2023-10-20T03:35:53.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080978099999999998 2023-10-20T03:35:53.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.512 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.448163+0000", 2023-10-20T03:35:53.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015406799999999999 2023-10-20T03:35:53.512 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.512 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.512 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.513 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.513 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:385 3.ds0 3:ba203010:::smithi17039300-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:35:53.513 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:35:53.513 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 23.000424228, 2023-10-20T03:35:53.513 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.074370642000000001, 2023-10-20T03:35:53.514 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.514 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:53.514 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:35:53.514 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:35:53.515 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:53.515 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 385 2023-10-20T03:35:53.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.515 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.516 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:35:53.516 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:35:53.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.518 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.518 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370862+0000", 2023-10-20T03:35:53.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2699999999999998e-07 2023-10-20T03:35:53.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370866+0000", 2023-10-20T03:35:53.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.42e-06 2023-10-20T03:35:53.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370880+0000", 2023-10-20T03:35:53.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3833e-05 2023-10-20T03:35:53.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370889+0000", 2023-10-20T03:35:53.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5900000000000008e-06 2023-10-20T03:35:53.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371887+0000", 2023-10-20T03:35:53.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00099814499999999998 2023-10-20T03:35:53.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2023-10-20T03:35:53.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.372069+0000", 2023-10-20T03:35:53.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018220899999999999 2023-10-20T03:35:53.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.525 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445044+0000", 2023-10-20T03:35:53.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072974686999999996 2023-10-20T03:35:53.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.525 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.525 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:53.526 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445137+0000", 2023-10-20T03:35:53.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3694000000000001e-05 2023-10-20T03:35:53.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445231+0000", 2023-10-20T03:35:53.527 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4136999999999997e-05 2023-10-20T03:35:53.527 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.527 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.528 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.528 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(osd.12.22:4 3.ds0 3:b8ff490e:::smithi17039300-34:head [assert-version v21,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:35:53.529 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:35:53.529 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 22.997975504999999, 2023-10-20T03:35:53.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083526739999999992, 2023-10-20T03:35:53.529 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.529 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:35:53.530 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:35:53.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:35:53.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373312+0000", 2023-10-20T03:35:53.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.424e-06 2023-10-20T03:35:53.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373326+0000", 2023-10-20T03:35:53.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4226000000000001e-05 2023-10-20T03:35:53.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373357+0000", 2023-10-20T03:35:53.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0698999999999999e-05 2023-10-20T03:35:53.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373388+0000", 2023-10-20T03:35:53.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0846000000000002e-05 2023-10-20T03:35:53.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.537 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373551+0000", 2023-10-20T03:35:53.537 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016350300000000001 2023-10-20T03:35:53.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.537 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:35:53.538 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373703+0000", 2023-10-20T03:35:53.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015216200000000001 2023-10-20T03:35:53.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.538 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.381662+0000", 2023-10-20T03:35:53.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079588139999999998 2023-10-20T03:35:53.539 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.539 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.540 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.540 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpReadReply(3.ds0 48/31 ECSubReadReply(tid=59, attrs_read=0))", 2023-10-20T03:35:53.540 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:35:53.541 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 22.995042987000001, 2023-10-20T03:35:53.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0055763890000000002, 2023-10-20T03:35:53.541 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.541 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:53.541 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.542 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:35:53.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:35:53.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376244+0000", 2023-10-20T03:35:53.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7570000000000001e-06 2023-10-20T03:35:53.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380643+0000", 2023-10-20T03:35:53.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00439931 2023-10-20T03:35:53.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380671+0000", 2023-10-20T03:35:53.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8019000000000001e-05 2023-10-20T03:35:53.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380685+0000", 2023-10-20T03:35:53.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4334e-05 2023-10-20T03:35:53.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.548 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.548 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380780+0000", 2023-10-20T03:35:53.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4087999999999998e-05 2023-10-20T03:35:53.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.549 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.549 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.381818+0000", 2023-10-20T03:35:53.550 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001038881 2023-10-20T03:35:53.550 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.550 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.551 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.551 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs2 48/47 ECSubWrite(tid=115, reqid=osd.13.0:115, at_version=48'27, trim_to=0'0, roll_forward_to=0'0))", 2023-10-20T03:35:53.552 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:35:53.552 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 22.981745114999999, 2023-10-20T03:35:53.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047760049999999998, 2023-10-20T03:35:53.552 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.552 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:53.553 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:35:53.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:35:53.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389541+0000", 2023-10-20T03:35:53.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5659999999999999e-06 2023-10-20T03:35:53.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389579+0000", 2023-10-20T03:35:53.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7070999999999998e-05 2023-10-20T03:35:53.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389605+0000", 2023-10-20T03:35:53.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6848000000000001e-05 2023-10-20T03:35:53.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389626+0000", 2023-10-20T03:35:53.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0579000000000001e-05 2023-10-20T03:35:53.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.560 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.560 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389781+0000", 2023-10-20T03:35:53.560 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015502300000000001 2023-10-20T03:35:53.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.561 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.561 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:53.561 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389807+0000", 2023-10-20T03:35:53.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6418999999999999e-05 2023-10-20T03:35:53.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:53.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394222+0000", 2023-10-20T03:35:53.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044143139999999999 2023-10-20T03:35:53.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.563 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.563 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394316+0000", 2023-10-20T03:35:53.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4184999999999994e-05 2023-10-20T03:35:53.564 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.564 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.564 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.565 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:35:53.565 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:35:53.565 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 22.941870002999998, 2023-10-20T03:35:53.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015918188, 2023-10-20T03:35:53.566 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.566 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:53.566 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:35:53.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:35:53.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429417+0000", 2023-10-20T03:35:53.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9429999999999999e-06 2023-10-20T03:35:53.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429432+0000", 2023-10-20T03:35:53.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4942e-05 2023-10-20T03:35:53.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429459+0000", 2023-10-20T03:35:53.572 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6740000000000001e-05 2023-10-20T03:35:53.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.572 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429472+0000", 2023-10-20T03:35:53.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3382e-05 2023-10-20T03:35:53.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.574 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.574 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445294+0000", 2023-10-20T03:35:53.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015821894999999999 2023-10-20T03:35:53.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445333+0000", 2023-10-20T03:35:53.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9286000000000003e-05 2023-10-20T03:35:53.576 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.576 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.576 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.576 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.577 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:35:53.577 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:35:53.577 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 22.94152102, 2023-10-20T03:35:53.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01562377, 2023-10-20T03:35:53.578 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.578 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:53.578 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.579 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.579 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:35:53.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.580 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.580 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:35:53.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429765+0000", 2023-10-20T03:35:53.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0079999999999999e-06 2023-10-20T03:35:53.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429772+0000", 2023-10-20T03:35:53.583 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2230000000000001e-06 2023-10-20T03:35:53.583 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.583 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.584 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429793+0000", 2023-10-20T03:35:53.584 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0707999999999999e-05 2023-10-20T03:35:53.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.584 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.585 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.585 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429806+0000", 2023-10-20T03:35:53.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3528e-05 2023-10-20T03:35:53.585 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.586 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.586 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.586 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445370+0000", 2023-10-20T03:35:53.586 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015563917 2023-10-20T03:35:53.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.587 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.587 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.587 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445388+0000", 2023-10-20T03:35:53.587 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7385999999999999e-05 2023-10-20T03:35:53.588 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.588 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.588 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.589 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:35:53.589 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:35:53.589 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 22.939870123999999, 2023-10-20T03:35:53.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014045581999999999, 2023-10-20T03:35:53.589 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.590 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:53.590 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.591 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:35:53.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.592 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.592 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.592 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:35:53.592 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.592 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.593 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.593 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431417+0000", 2023-10-20T03:35:53.593 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4159999999999998e-06 2023-10-20T03:35:53.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.594 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.594 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.594 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431433+0000", 2023-10-20T03:35:53.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5653000000000001e-05 2023-10-20T03:35:53.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.595 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.595 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431472+0000", 2023-10-20T03:35:53.596 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9232e-05 2023-10-20T03:35:53.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.596 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.596 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.596 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431492+0000", 2023-10-20T03:35:53.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9550000000000001e-05 2023-10-20T03:35:53.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445435+0000", 2023-10-20T03:35:53.598 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01394307 2023-10-20T03:35:53.598 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.598 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.598 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.599 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445460+0000", 2023-10-20T03:35:53.599 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5661e-05 2023-10-20T03:35:53.599 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.599 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.600 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.600 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:35:53.600 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:35:53.601 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 22.939064124000001, 2023-10-20T03:35:53.601 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013263258999999999, 2023-10-20T03:35:53.601 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.601 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:53.602 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.602 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.602 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.602 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:35:53.603 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.603 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.603 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:35:53.604 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.604 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.604 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.605 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432228+0000", 2023-10-20T03:35:53.605 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.08e-06 2023-10-20T03:35:53.605 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.605 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.605 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.606 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432236+0000", 2023-10-20T03:35:53.606 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6629999999999993e-06 2023-10-20T03:35:53.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432260+0000", 2023-10-20T03:35:53.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3998999999999998e-05 2023-10-20T03:35:53.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432272+0000", 2023-10-20T03:35:53.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2023e-05 2023-10-20T03:35:53.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445470+0000", 2023-10-20T03:35:53.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013198507 2023-10-20T03:35:53.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445484+0000", 2023-10-20T03:35:53.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3987e-05 2023-10-20T03:35:53.610 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.610 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.611 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.611 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:35:53.611 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:35:53.612 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 22.938759775000001, 2023-10-20T03:35:53.612 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012997794, 2023-10-20T03:35:53.612 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.612 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:53.613 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.613 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.613 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:35:53.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:35:53.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.615 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432526+0000", 2023-10-20T03:35:53.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1400000000000001e-06 2023-10-20T03:35:53.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.617 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432532+0000", 2023-10-20T03:35:53.617 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.823e-06 2023-10-20T03:35:53.617 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.617 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.617 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.618 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432552+0000", 2023-10-20T03:35:53.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0032999999999999e-05 2023-10-20T03:35:53.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432567+0000", 2023-10-20T03:35:53.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4932999999999999e-05 2023-10-20T03:35:53.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.620 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.620 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445506+0000", 2023-10-20T03:35:53.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012939315 2023-10-20T03:35:53.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.620 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.621 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.621 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445523+0000", 2023-10-20T03:35:53.621 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6549999999999999e-05 2023-10-20T03:35:53.621 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.622 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.622 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.623 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs2 49/47 ECSubWrite(tid=118, reqid=osd.13.0:118, at_version=49'29, trim_to=0'0, roll_forward_to=48'27))", 2023-10-20T03:35:53.623 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:35:53.623 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.960426544000001, 2023-10-20T03:35:53.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043378410000000003, 2023-10-20T03:35:53.624 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.624 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:53.624 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.624 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:35:53.625 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.625 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.626 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:35:53.626 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.627 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.627 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410859+0000", 2023-10-20T03:35:53.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0699999999999999e-07 2023-10-20T03:35:53.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410884+0000", 2023-10-20T03:35:53.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.529e-05 2023-10-20T03:35:53.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410898+0000", 2023-10-20T03:35:53.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.383e-05 2023-10-20T03:35:53.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.630 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.630 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410928+0000", 2023-10-20T03:35:53.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0329e-05 2023-10-20T03:35:53.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.631 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.411037+0000", 2023-10-20T03:35:53.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010869099999999999 2023-10-20T03:35:53.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:35:53.632 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.411048+0000", 2023-10-20T03:35:53.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0905e-05 2023-10-20T03:35:53.633 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.633 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:35:53.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415159+0000", 2023-10-20T03:35:53.634 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041115520000000001 2023-10-20T03:35:53.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.634 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.635 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415196+0000", 2023-10-20T03:35:53.635 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7036999999999998e-05 2023-10-20T03:35:53.635 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.635 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.635 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.636 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.636 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpRead(3.cs2 55/54 ECSubRead(tid=70, to_read={3:3382ea17:::smithi17039300-35:6=0,262144,0}, subchunks={3:3382ea17:::smithi17039300-35:6=[0,1]}, attrs_to_read=))", 2023-10-20T03:35:53.636 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:37.459240+0000", 2023-10-20T03:35:53.636 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.912045438, 2023-10-20T03:35:53.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044608460000000001, 2023-10-20T03:35:53.637 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:35:53.637 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:35:53.637 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:35:53.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:35:53.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459240+0000", 2023-10-20T03:35:53.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.639 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:35:53.639 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459240+0000", 2023-10-20T03:35:53.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:35:53.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.640 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:35:53.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459241+0000", 2023-10-20T03:35:53.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.84e-06 2023-10-20T03:35:53.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:35:53.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459256+0000", 2023-10-20T03:35:53.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.454e-05 2023-10-20T03:35:53.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:35:53.642 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459288+0000", 2023-10-20T03:35:53.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2329000000000002e-05 2023-10-20T03:35:53.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.643 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:35:53.644 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459307+0000", 2023-10-20T03:35:53.644 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9046999999999999e-05 2023-10-20T03:35:53.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:35:53.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459460+0000", 2023-10-20T03:35:53.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015253200000000001 2023-10-20T03:35:53.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:35:53.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:35:53.646 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:35:53.646 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.463700+0000", 2023-10-20T03:35:53.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042405580000000002 2023-10-20T03:35:53.647 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.647 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.647 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.647 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:35:53.648 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:35:53.648 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:35:53.650 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:53.651 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.652 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2023-10-20T03:35:52.947236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67961376,"num_objects":71,"num_object_clones":25,"num_object_copies":487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":4,"num_read":395,"num_read_kb":77480,"num_write":754,"num_write_kb":80692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":17154048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":741,"ondisk_log_size":741,"up":129,"acting":129,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":522144,"kb_used_data":116752,"kb_used_omap":23,"kb_used_meta":405096,"kb_avail":1415055456,"statfs":{"total":1449551462400,"available":1449016786944,"internally_reserved":0,"allocated":119554048,"data_stored":134202050,"data_compressed":1291040,"data_compressed_allocated":2711552,"data_compressed_original":6316032,"omap_allocated":23908,"internal_metadata":414818972},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1622016,"num_objects":-2,"num_object_clones":-2,"num_object_copies":-14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":5,"num_read_kb":9889,"num_write":1,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043055"},"pg_stats":[{"pgid":"3.f","version":"65'60","reported_seq":222,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-20T03:35:50.931112+0000","last_change":"2023-10-20T03:35:50.261325+0000","last_active":"2023-10-20T03:35:50.931112+0000","last_peered":"2023-10-20T03:35:50.931112+0000","last_clean":"2023-10-20T03:35:50.931112+0000","last_became_active":"2023-10-20T03:35:30.382595+0000","last_became_peered":"2023-10-20T03:35:30.382595+0000","last_unstale":"2023-10-20T03:35:50.931112+0000","last_undegraded":"2023-10-20T03:35:50.931112+0000","last_fullsized":"2023-10-20T03:35:50.931112+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:16.773345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001141478,"stat_sum":{"num_bytes":6553600,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":43,"num_write_kb":7232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,15,14,2,11,1],"acting":[0,3,15,14,2,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"}]},{"pgid":"3.e","version":"61'23","reported_seq":122,"reported_epoch":69,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:50.942848+0000","last_change":"2023-10-20T03:35:16.262200+0000","last_active":"2023-10-20T03:35:50.942848+0000","last_peered":"2023-10-20T03:35:50.942848+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:14.230273+0000","last_became_peered":"2023-10-20T03:35:14.230273+0000","last_unstale":"2023-10-20T03:35:50.942848+0000","last_undegraded":"2023-10-20T03:35:16.261909+0000","last_fullsized":"2023-10-20T03:35:14.212645+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:42.427413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3375104,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":27,"num_read_kb":7954,"num_write":61,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,2147483647,4,13,3,5],"acting":[11,14,2147483647,4,13,3,5],"avail_no_missing":["11(0)","3(5)","4(3)","5(6)","13(4)","14(1)"],"object_location_counts":[{"shards":"3(5),4(3),5(6),11(0),13(4),14(1)","objects":7}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"61'51","reported_seq":210,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-20T03:35:50.937572+0000","last_change":"2023-10-20T03:35:49.924767+0000","last_active":"2023-10-20T03:35:50.937572+0000","last_peered":"2023-10-20T03:35:50.937572+0000","last_clean":"2023-10-20T03:35:50.937572+0000","last_became_active":"2023-10-20T03:35:38.820900+0000","last_became_peered":"2023-10-20T03:35:38.820900+0000","last_unstale":"2023-10-20T03:35:50.937572+0000","last_undegraded":"2023-10-20T03:35:50.937572+0000","last_fullsized":"2023-10-20T03:35:50.937572+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:26.236589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015658110000000001,"stat_sum":{"num_bytes":7503872,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":2169,"num_write":24,"num_write_kb":8672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,11,0,8,13,5],"acting":[1,4,11,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"}]},{"pgid":"3.c","version":"69'56","reported_seq":227,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-20T03:35:50.930197+0000","last_change":"2023-10-20T03:35:50.264539+0000","last_active":"2023-10-20T03:35:50.930197+0000","last_peered":"2023-10-20T03:35:50.930197+0000","last_clean":"2023-10-20T03:35:50.930197+0000","last_became_active":"2023-10-20T03:35:38.820502+0000","last_became_peered":"2023-10-20T03:35:38.820502+0000","last_unstale":"2023-10-20T03:35:50.930197+0000","last_undegraded":"2023-10-20T03:35:50.930197+0000","last_fullsized":"2023-10-20T03:35:50.930197+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:39.346969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018693259999999999,"stat_sum":{"num_bytes":573440,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":7062,"num_write":19,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":573440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,5,3,7,1,9],"acting":[0,13,5,3,7,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"}]},{"pgid":"3.b","version":"61'34","reported_seq":212,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-20T03:35:50.929946+0000","last_change":"2023-10-20T03:35:50.263126+0000","last_active":"2023-10-20T03:35:50.929946+0000","last_peered":"2023-10-20T03:35:50.929946+0000","last_clean":"2023-10-20T03:35:50.929946+0000","last_became_active":"2023-10-20T03:35:38.816060+0000","last_became_peered":"2023-10-20T03:35:38.816060+0000","last_unstale":"2023-10-20T03:35:50.929946+0000","last_undegraded":"2023-10-20T03:35:50.929946+0000","last_fullsized":"2023-10-20T03:35:50.929946+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:24.722480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013173530000000001,"stat_sum":{"num_bytes":5488640,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":17,"num_write_kb":5760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2752512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,10,0,11,4,14],"acting":[13,1,10,0,11,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"}]},{"pgid":"3.a","version":"61'25","reported_seq":111,"reported_epoch":69,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:50.939176+0000","last_change":"2023-10-20T03:35:38.812783+0000","last_active":"2023-10-20T03:35:50.939176+0000","last_peered":"2023-10-20T03:35:50.939176+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:38.812783+0000","last_became_peered":"2023-10-20T03:35:38.812783+0000","last_unstale":"2023-10-20T03:35:50.939176+0000","last_undegraded":"2023-10-20T03:35:38.799500+0000","last_fullsized":"2023-10-20T03:35:38.798624+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:13.655430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4472832,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":3895,"num_write":40,"num_write_kb":4672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2147483647,10,3,15,8],"acting":[9,4,2147483647,10,3,15,8],"avail_no_missing":["9(0)","3(4)","4(1)","8(6)","10(3)","15(5)"],"object_location_counts":[{"shards":"3(4),4(1),8(6),9(0),10(3),15(5)","objects":7}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"60'34","reported_seq":188,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-20T03:35:50.935687+0000","last_change":"2023-10-20T03:35:50.260924+0000","last_active":"2023-10-20T03:35:50.935687+0000","last_peered":"2023-10-20T03:35:50.935687+0000","last_clean":"2023-10-20T03:35:50.935687+0000","last_became_active":"2023-10-20T03:35:14.780099+0000","last_became_peered":"2023-10-20T03:35:14.780099+0000","last_unstale":"2023-10-20T03:35:50.935687+0000","last_undegraded":"2023-10-20T03:35:50.935687+0000","last_fullsized":"2023-10-20T03:35:50.935687+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:22.486174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093430599999999998,"stat_sum":{"num_bytes":2211840,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":6488,"num_write":19,"num_write_kb":4384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,10,11,4,9,3],"acting":[7,2,10,11,4,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"}]},{"pgid":"3.8","version":"61'12","reported_seq":100,"reported_epoch":69,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:50.937470+0000","last_change":"2023-10-20T03:35:16.333457+0000","last_active":"2023-10-20T03:35:50.937470+0000","last_peered":"2023-10-20T03:35:50.937470+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:14.226955+0000","last_became_peered":"2023-10-20T03:35:14.226955+0000","last_unstale":"2023-10-20T03:35:50.937470+0000","last_undegraded":"2023-10-20T03:35:16.333026+0000","last_fullsized":"2023-10-20T03:35:14.202565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:33.291635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4128768,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":12102,"num_write":11,"num_write_kb":4032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,1,11,3,5,0,7],"acting":[2147483647,1,11,3,5,0,7],"avail_no_missing":["1(1)","0(5)","3(3)","5(4)","7(6)","11(2)"],"object_location_counts":[{"shards":"0(5),1(1),3(3),5(4),7(6),11(2)","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"26'157","reported_seq":250,"reported_epoch":68,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:50.261463+0000","last_change":"2023-10-20T03:35:12.183237+0000","last_active":"2023-10-20T03:35:50.261463+0000","last_peered":"2023-10-20T03:35:50.261463+0000","last_clean":"2023-10-20T03:35:05.091025+0000","last_became_active":"2023-10-20T03:35:12.183237+0000","last_became_peered":"2023-10-20T03:35:12.183237+0000","last_unstale":"2023-10-20T03:35:50.261463+0000","last_undegraded":"2023-10-20T03:35:12.182136+0000","last_fullsized":"2023-10-20T03:35:12.181940+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_clean_scrub_stamp":"2023-10-20T03:35:01.005741+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:23.418644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15],"acting":[15],"avail_no_missing":["15"],"object_location_counts":[{"shards":"15","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-20T03:35:48.907646+0000","last_change":"2023-10-20T03:35:05.117915+0000","last_active":"2023-10-20T03:35:48.907646+0000","last_peered":"2023-10-20T03:35:48.907646+0000","last_clean":"2023-10-20T03:35:48.907646+0000","last_became_active":"2023-10-20T03:35:05.117258+0000","last_became_peered":"2023-10-20T03:35:05.117258+0000","last_unstale":"2023-10-20T03:35:48.907646+0000","last_undegraded":"2023-10-20T03:35:48.907646+0000","last_fullsized":"2023-10-20T03:35:48.907646+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:32.359335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"66'41","reported_seq":201,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-20T03:35:50.275731+0000","last_change":"2023-10-20T03:35:50.276644+0000","last_active":"2023-10-20T03:35:50.275731+0000","last_peered":"2023-10-20T03:35:50.275731+0000","last_clean":"2023-10-20T03:35:50.275731+0000","last_became_active":"2023-10-20T03:35:14.221482+0000","last_became_peered":"2023-10-20T03:35:14.221482+0000","last_unstale":"2023-10-20T03:35:50.275731+0000","last_undegraded":"2023-10-20T03:35:50.275731+0000","last_fullsized":"2023-10-20T03:35:50.275731+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:17.151227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017396709999999999,"stat_sum":{"num_bytes":3588096,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":12493,"num_write":81,"num_write_kb":3504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,0,4,8,10,7],"acting":[14,3,0,4,8,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"}]},{"pgid":"2.0","version":"0'0","reported_seq":50,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-20T03:35:49.438560+0000","last_change":"2023-10-20T03:35:05.201208+0000","last_active":"2023-10-20T03:35:49.438560+0000","last_peered":"2023-10-20T03:35:49.438560+0000","last_clean":"2023-10-20T03:35:49.438560+0000","last_became_active":"2023-10-20T03:35:05.200512+0000","last_became_peered":"2023-10-20T03:35:05.200512+0000","last_unstale":"2023-10-20T03:35:49.438560+0000","last_undegraded":"2023-10-20T03:35:49.438560+0000","last_fullsized":"2023-10-20T03:35:49.438560+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:30.668122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"68'72","reported_seq":220,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-10-20T03:35:50.930254+0000","last_change":"2023-10-20T03:35:50.273399+0000","last_active":"2023-10-20T03:35:50.930254+0000","last_peered":"2023-10-20T03:35:50.930254+0000","last_clean":"2023-10-20T03:35:50.930254+0000","last_became_active":"2023-10-20T03:35:14.311165+0000","last_became_peered":"2023-10-20T03:35:14.311165+0000","last_unstale":"2023-10-20T03:35:50.930254+0000","last_undegraded":"2023-10-20T03:35:50.930254+0000","last_fullsized":"2023-10-20T03:35:50.930254+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:35.993486+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.011807401,"stat_sum":{"num_bytes":5799936,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":1352,"num_write":25,"num_write_kb":6032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,14,10,7,3,2],"acting":[13,4,14,10,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"}]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-20T03:35:48.907283+0000","last_change":"2023-10-20T03:35:05.113211+0000","last_active":"2023-10-20T03:35:48.907283+0000","last_peered":"2023-10-20T03:35:48.907283+0000","last_clean":"2023-10-20T03:35:48.907283+0000","last_became_active":"2023-10-20T03:35:05.112592+0000","last_became_peered":"2023-10-20T03:35:05.112592+0000","last_unstale":"2023-10-20T03:35:48.907283+0000","last_undegraded":"2023-10-20T03:35:48.907283+0000","last_fullsized":"2023-10-20T03:35:48.907283+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:27.383114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"69'41","reported_seq":133,"reported_epoch":69,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:50.983902+0000","last_change":"2023-10-20T03:35:16.236066+0000","last_active":"2023-10-20T03:35:50.983902+0000","last_peered":"2023-10-20T03:35:50.983902+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:14.779378+0000","last_became_peered":"2023-10-20T03:35:14.779378+0000","last_unstale":"2023-10-20T03:35:50.983902+0000","last_undegraded":"2023-10-20T03:35:16.227298+0000","last_fullsized":"2023-10-20T03:35:14.210627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:20.560272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9846784,"num_objects":10,"num_object_clones":7,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":20,"num_read_kb":2361,"num_write":31,"num_write_kb":9616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4,2,2147483647,15,9],"acting":[3,7,4,2,2147483647,15,9],"avail_no_missing":["3(0)","2(3)","4(2)","7(1)","9(6)","15(5)"],"object_location_counts":[{"shards":"2(3),3(0),4(2),7(1),9(6),15(5)","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":50,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-20T03:35:49.438836+0000","last_change":"2023-10-20T03:35:05.114672+0000","last_active":"2023-10-20T03:35:49.438836+0000","last_peered":"2023-10-20T03:35:49.438836+0000","last_clean":"2023-10-20T03:35:49.438836+0000","last_became_active":"2023-10-20T03:35:05.114006+0000","last_became_peered":"2023-10-20T03:35:05.114006+0000","last_unstale":"2023-10-20T03:35:49.438836+0000","last_undegraded":"2023-10-20T03:35:49.438836+0000","last_fullsized":"2023-10-20T03:35:49.438836+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:28.429081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"60'38","reported_seq":220,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-20T03:35:50.262996+0000","last_change":"2023-10-20T03:35:50.264108+0000","last_active":"2023-10-20T03:35:50.262996+0000","last_peered":"2023-10-20T03:35:50.262996+0000","last_clean":"2023-10-20T03:35:50.262996+0000","last_became_active":"2023-10-20T03:35:38.811204+0000","last_became_peered":"2023-10-20T03:35:38.811204+0000","last_unstale":"2023-10-20T03:35:50.262996+0000","last_undegraded":"2023-10-20T03:35:50.262996+0000","last_fullsized":"2023-10-20T03:35:50.262996+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:22.169932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017028810000000001,"stat_sum":{"num_bytes":2998272,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":1014,"num_write":16,"num_write_kb":3600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4866048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,1,0,11,2,14],"acting":[10,8,1,0,11,2,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"}]},{"pgid":"3.5","version":"60'30","reported_seq":184,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-20T03:35:50.135821+0000","last_change":"2023-10-20T03:35:50.136196+0000","last_active":"2023-10-20T03:35:50.135821+0000","last_peered":"2023-10-20T03:35:50.135821+0000","last_clean":"2023-10-20T03:35:50.135821+0000","last_became_active":"2023-10-20T03:35:14.779482+0000","last_became_peered":"2023-10-20T03:35:14.779482+0000","last_unstale":"2023-10-20T03:35:50.135821+0000","last_undegraded":"2023-10-20T03:35:50.135821+0000","last_fullsized":"2023-10-20T03:35:50.135821+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:28.882707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058458399999999997,"stat_sum":{"num_bytes":802816,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":3604,"num_write":29,"num_write_kb":1664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,2,9,1,11],"acting":[8,13,5,2,9,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-20T03:35:49.923194+0000","last_change":"2023-10-20T03:35:05.114312+0000","last_active":"2023-10-20T03:35:49.923194+0000","last_peered":"2023-10-20T03:35:49.923194+0000","last_clean":"2023-10-20T03:35:49.923194+0000","last_became_active":"2023-10-20T03:35:05.113456+0000","last_became_peered":"2023-10-20T03:35:05.113456+0000","last_unstale":"2023-10-20T03:35:49.923194+0000","last_undegraded":"2023-10-20T03:35:49.923194+0000","last_fullsized":"2023-10-20T03:35:49.923194+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:22.341298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"51'4","reported_seq":85,"reported_epoch":69,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:50.938318+0000","last_change":"2023-10-20T03:35:16.272112+0000","last_active":"2023-10-20T03:35:50.938318+0000","last_peered":"2023-10-20T03:35:50.938318+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:14.778571+0000","last_became_peered":"2023-10-20T03:35:14.778571+0000","last_unstale":"2023-10-20T03:35:50.938318+0000","last_undegraded":"2023-10-20T03:35:16.266905+0000","last_fullsized":"2023-10-20T03:35:14.202765+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:18.274609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98304,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":6,"num_read_kb":290,"num_write":3,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,15,9,2,2147483647,10,13],"acting":[1,15,9,2,2147483647,10,13],"avail_no_missing":["1(0)","2(3)","9(2)","10(5)","13(6)","15(1)"],"object_location_counts":[{"shards":"1(0),2(3),9(2),10(5),13(6),15(1)","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":50,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-20T03:35:50.259855+0000","last_change":"2023-10-20T03:35:05.122638+0000","last_active":"2023-10-20T03:35:50.259855+0000","last_peered":"2023-10-20T03:35:50.259855+0000","last_clean":"2023-10-20T03:35:50.259855+0000","last_became_active":"2023-10-20T03:35:05.122056+0000","last_became_peered":"2023-10-20T03:35:05.122056+0000","last_unstale":"2023-10-20T03:35:50.259855+0000","last_undegraded":"2023-10-20T03:35:50.259855+0000","last_fullsized":"2023-10-20T03:35:50.259855+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:21.001893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"60'11","reported_seq":96,"reported_epoch":69,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:50.931073+0000","last_change":"2023-10-20T03:35:16.265401+0000","last_active":"2023-10-20T03:35:50.931073+0000","last_peered":"2023-10-20T03:35:50.931073+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:14.228865+0000","last_became_peered":"2023-10-20T03:35:14.228865+0000","last_unstale":"2023-10-20T03:35:50.931073+0000","last_undegraded":"2023-10-20T03:35:16.263346+0000","last_fullsized":"2023-10-20T03:35:14.206401+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:34.878903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1818624,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":6,"num_read_kb":3,"num_write":10,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,2147483647,1,14,10,0],"acting":[13,11,2147483647,1,14,10,0],"avail_no_missing":["13(0)","0(6)","1(3)","10(5)","11(1)","14(4)"],"object_location_counts":[{"shards":"0(6),1(3),10(5),11(1),13(0),14(4)","objects":3}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-10-20T03:35:50.261958+0000","last_change":"2023-10-20T03:35:05.204140+0000","last_active":"2023-10-20T03:35:50.261958+0000","last_peered":"2023-10-20T03:35:50.261958+0000","last_clean":"2023-10-20T03:35:50.261958+0000","last_became_active":"2023-10-20T03:35:05.203380+0000","last_became_peered":"2023-10-20T03:35:05.203380+0000","last_unstale":"2023-10-20T03:35:50.261958+0000","last_undegraded":"2023-10-20T03:35:50.261958+0000","last_fullsized":"2023-10-20T03:35:50.261958+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:16.855327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"61'52","reported_seq":186,"reported_epoch":69,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:35:50.930516+0000","last_change":"2023-10-20T03:35:38.815134+0000","last_active":"2023-10-20T03:35:50.930516+0000","last_peered":"2023-10-20T03:35:50.930516+0000","last_clean":"2023-10-20T03:35:13.171293+0000","last_became_active":"2023-10-20T03:35:38.815134+0000","last_became_peered":"2023-10-20T03:35:38.815134+0000","last_unstale":"2023-10-20T03:35:50.930516+0000","last_undegraded":"2023-10-20T03:35:38.802376+0000","last_fullsized":"2023-10-20T03:35:38.801503+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:18.927740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8110080,"num_objects":14,"num_object_clones":9,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":28,"num_read_kb":16563,"num_write":122,"num_write_kb":8448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":8962048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14,2147483647,2,8,0,15],"acting":[13,14,2147483647,2,8,0,15],"avail_no_missing":["13(0)","0(5)","2(3)","8(4)","14(1)","15(6)"],"object_location_counts":[{"shards":"0(5),2(3),8(4),13(0),14(1),15(6)","objects":14}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":50,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-10-20T03:35:49.439016+0000","last_change":"2023-10-20T03:35:05.199365+0000","last_active":"2023-10-20T03:35:49.439016+0000","last_peered":"2023-10-20T03:35:49.439016+0000","last_clean":"2023-10-20T03:35:49.439016+0000","last_became_active":"2023-10-20T03:35:05.198615+0000","last_became_peered":"2023-10-20T03:35:05.198615+0000","last_unstale":"2023-10-20T03:35:49.439016+0000","last_undegraded":"2023-10-20T03:35:49.439016+0000","last_fullsized":"2023-10-20T03:35:49.439016+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:23.667783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":67371008,"num_objects":69,"num_object_clones":25,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":46,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":4,"num_read":259,"num_read_kb":77362,"num_write":551,"num_write_kb":77232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":17154048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":112889856,"data_stored":130953216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":584,"ondisk_log_size":584,"up":112,"acting":112,"num_store_stats":15},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":15078,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280525,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35964,"kb_used_data":8940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335876,"statfs":{"total":96636764160,"available":96599937024,"internally_reserved":0,"allocated":9154560,"data_stored":10168423,"data_compressed":90917,"data_compressed_allocated":188416,"data_compressed_original":376832,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.46}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.748}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.956}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8120000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3439999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4510000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4940000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.198}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2949999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.403}]}]},{"osd":15,"up_from":22,"seq":94489280526,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34540,"kb_used_data":7500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337300,"statfs":{"total":96636764160,"available":96601395200,"internally_reserved":0,"allocated":7680000,"data_stored":10615431,"data_compressed":98456,"data_compressed_allocated":221184,"data_compressed_original":888832,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]}]},{"osd":14,"up_from":22,"seq":94489280526,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36980,"kb_used_data":9956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334860,"statfs":{"total":96636764160,"available":96598896640,"internally_reserved":0,"allocated":10194944,"data_stored":11208807,"data_compressed":90917,"data_compressed_allocated":188416,"data_compressed_original":376832,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,7,8,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.077}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4470000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.341}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]}]},{"osd":5,"up_from":21,"seq":90194313230,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":4841472,"data_stored":4935052,"data_compressed":88910,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,7,8,9,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1579999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.504}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280525,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37720,"kb_used_data":10696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334120,"statfs":{"total":96636764160,"available":96598138880,"internally_reserved":0,"allocated":10952704,"data_stored":11909223,"data_compressed":90917,"data_compressed_allocated":188416,"data_compressed_original":376832,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,5,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.839}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7929999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.681}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3500000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6120000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2250000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.679}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0550000000000002}]}]},{"osd":1,"up_from":22,"seq":94489280525,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35080,"kb_used_data":8060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336760,"statfs":{"total":96636764160,"available":96600842240,"internally_reserved":0,"allocated":8253440,"data_stored":8153191,"data_compressed":90917,"data_compressed_allocated":188416,"data_compressed_original":376832,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.673}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.766}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.008}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9750000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.923}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.085}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7000000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7890000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4119999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5649999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.056}]}]},{"osd":2,"up_from":22,"seq":94489280525,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36264,"kb_used_data":9240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335576,"statfs":{"total":96636764160,"available":96599629824,"internally_reserved":0,"allocated":9461760,"data_stored":11823207,"data_compressed":90917,"data_compressed_allocated":188416,"data_compressed_original":376832,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,7,8,9,10,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.226}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4390000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.48}]}]},{"osd":3,"up_from":22,"seq":94489280525,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37256,"kb_used_data":10236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334584,"statfs":{"total":96636764160,"available":96598614016,"internally_reserved":0,"allocated":10481664,"data_stored":12242902,"data_compressed":88910,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,7,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5609999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.653}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6060000000000001}]}]},{"osd":4,"up_from":22,"seq":94489280526,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37696,"kb_used_data":10672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334144,"statfs":{"total":96636764160,"available":96598163456,"internally_reserved":0,"allocated":10928128,"data_stored":12519527,"data_compressed":90917,"data_compressed_allocated":188416,"data_compressed_original":376832,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.391}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.341}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2390000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.407}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]}]},{"osd":6,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":323584,"data_stored":647661,"data_compressed":18691,"data_compressed_allocated":57344,"data_compressed_original":561152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,3,5,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4139999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7290000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7989999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.738}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9089999999999998}]}]},{"osd":7,"up_from":22,"seq":94489280524,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":6720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":6881280,"data_stored":8355798,"data_compressed":88910,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,2,3,4,5,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.077}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.24}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8420000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]}]},{"osd":8,"up_from":22,"seq":94489280525,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35492,"kb_used_data":8472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336348,"statfs":{"total":96636764160,"available":96600420352,"internally_reserved":0,"allocated":8675328,"data_stored":8544214,"data_compressed":88910,"data_compressed_allocated":184320,"data_compressed_original":368640,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,5,7,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.266}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.369}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.609}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6950000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.883}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8520000000000001}]}]},{"osd":9,"up_from":22,"seq":94489280525,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31780,"kb_used_data":4752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340060,"statfs":{"total":96636764160,"available":96604221440,"internally_reserved":0,"allocated":4866048,"data_stored":6260839,"data_compressed":90917,"data_compressed_allocated":188416,"data_compressed_original":376832,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[2,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.163}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]}]},{"osd":10,"up_from":22,"seq":94489280524,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34232,"kb_used_data":7208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337608,"statfs":{"total":96636764160,"available":96601710592,"internally_reserved":0,"allocated":7380992,"data_stored":7247975,"data_compressed":90917,"data_compressed_allocated":188416,"data_compressed_original":376832,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,8,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3959999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.252}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.423}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.407}]}]},{"osd":11,"up_from":22,"seq":94489280526,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36280,"kb_used_data":9256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335560,"statfs":{"total":96636764160,"available":96599613440,"internally_reserved":0,"allocated":9478144,"data_stored":9569800,"data_compressed":90917,"data_compressed_allocated":188416,"data_compressed_original":376832,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2330000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.776}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]}]},{"osd":12,"up_from":0,"seq":94489280524,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10190848,"data_stored":11378688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7491584,"data_stored":7622656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9105408,"data_stored":11698176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10137600,"data_stored":12091392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":10571776,"data_stored":12394496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096000,"data_stored":4415488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6537216,"data_stored":8204288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":7925760,"data_stored":8024064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4509696,"data_stored":6135808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6619136,"data_stored":6717440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8720384,"data_stored":9039872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":1888256,"data_stored":2953216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":9637888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":9433088,"data_stored":10678272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":7270400,"data_stored":9961472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T03:35:53.656 INFO:tasks.thrashosds.thrasher:Setting 3.6 to [9, 13, 1, 12, 5, 10, 8, 4, 11, 15, 3, 2, 7, 6] 2023-10-20T03:35:53.656 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.6', '9', '13', '1', '12', '5', '10', '8', '4', '11', '15', '3', '2', '7', '6'] 2023-10-20T03:35:53.657 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.6 9 13 1 12 5 10 8 4 11 15 3 2 7 6 2023-10-20T03:35:53.658 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-10-20T03:35:53.659 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:53.659 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.659 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:53.660 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.660 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:53.661 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-10-20T03:35:53.662 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:53.648+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:53.663 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:35:53.661+0000 7f5563fff640 0 -- 172.21.15.33:0/4081491672 >> [v2:172.21.15.170:6819/826332016,v1:172.21.15.170:6822/826332016] conn(0x7f556c21e930 msgr2=0x7f556c220d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-10-20T03:35:53.668 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:53.674 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:53.675 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:53.751 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:53.749+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:53.796 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:53.797 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:53.811 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-10-20T03:35:53.858 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:53.856+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:53.930 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:53.931 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:53.945 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-10-20T03:35:53.952 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:53.949+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:53.964 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2023-10-20T03:35:53.971 INFO:tasks.rados.rados.0.smithi170.stdout:247: done (2 left) 2023-10-20T03:35:53.971 INFO:tasks.rados.rados.0.smithi170.stdout:248: done (1 left) 2023-10-20T03:35:53.971 INFO:tasks.rados.rados.0.smithi170.stdout:249: done (0 left) 2023-10-20T03:35:53.971 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 17 current snap is 19 2023-10-20T03:35:53.972 INFO:tasks.rados.rados.0.smithi170.stdout:250: seq_num 114 ranges {278528=753664,1032192=688128,1720320=475136} 2023-10-20T03:35:53.985 INFO:tasks.rados.rados.0.smithi170.stdout:250: writing smithi17039300-17 from 278528 to 1032192 tid 1 2023-10-20T03:35:53.994 INFO:tasks.rados.rados.0.smithi170.stdout:250: writing smithi17039300-17 from 1032192 to 1720320 tid 2 2023-10-20T03:35:54.000 INFO:tasks.rados.rados.0.smithi170.stdout:250: writing smithi17039300-17 from 1720320 to 2195456 tid 3 2023-10-20T03:35:54.001 INFO:tasks.rados.rados.0.smithi170.stdout:251: snap_create 2023-10-20T03:35:54.023 INFO:tasks.rados.rados.0.smithi170.stdout:250: finishing write tid 1 to smithi17039300-17 2023-10-20T03:35:54.025 INFO:tasks.rados.rados.0.smithi170.stdout:250: finishing write tid 2 to smithi17039300-17 2023-10-20T03:35:54.026 INFO:tasks.rados.rados.0.smithi170.stdout:250: finishing write tid 3 to smithi17039300-17 2023-10-20T03:35:54.027 INFO:tasks.rados.rados.0.smithi170.stdout:250: finishing write tid 4 to smithi17039300-17 2023-10-20T03:35:54.030 INFO:tasks.rados.rados.0.smithi170.stdout:250: finishing write tid 5 to smithi17039300-17 2023-10-20T03:35:54.030 INFO:tasks.rados.rados.0.smithi170.stdout:250: oid 17 updating version 0 to 54 2023-10-20T03:35:54.031 INFO:tasks.rados.rados.0.smithi170.stdout:250: oid 17 updating version 54 to 57 2023-10-20T03:35:54.031 INFO:tasks.rados.rados.0.smithi170.stdout:250: oid 17 version 57 is already newer than 56 2023-10-20T03:35:54.031 INFO:tasks.rados.rados.0.smithi170.stdout:250: oid 17 version 57 is already newer than 55 2023-10-20T03:35:54.031 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 57 (ObjNum 114 snap 19 seq_num 114) dirty exists 2023-10-20T03:35:54.032 INFO:tasks.rados.rados.0.smithi170.stdout:250: left oid 17 (ObjNum 114 snap 19 seq_num 114) 2023-10-20T03:35:54.053 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:54.048+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:54.117 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:54.117 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:54.133 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-10-20T03:35:54.155 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:35:54.150+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:35:54.256 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:54.253+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:54.303 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:54.303 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:54.317 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-10-20T03:35:54.356 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:35:54.353+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:35:54.442 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:54.443 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:54.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-10-20T03:35:54.459 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:54.456+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:54.558 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:54.554+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:54.585 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:54.586 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:54.600 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-10-20T03:35:54.659 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:54.656+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:54.726 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:54.726 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:54.740 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-10-20T03:35:54.759 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:54.756+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:54.860 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:54.860+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:54.861 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:54.861 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:54.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-10-20T03:35:54.961 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:54.956+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:54.975 INFO:teuthology.orchestra.run.smithi033.stderr:set 3.6 pg_upmap_items mapping to [9->13,1->12,5->10,8->4,11->15,3->2,7->6] 2023-10-20T03:35:54.983 INFO:tasks.rados.rados.0.smithi170.stdout:250: done (1 left) 2023-10-20T03:35:54.983 INFO:tasks.rados.rados.0.smithi170.stdout:251: done (0 left) 2023-10-20T03:35:54.984 INFO:tasks.rados.rados.0.smithi170.stdout:252: snap_create 2023-10-20T03:35:54.997 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:54.997 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:55.012 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-10-20T03:35:55.062 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:55.060+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:55.163 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:55.161+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:55.175 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:55.175 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:55.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-10-20T03:35:55.264 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:55.261+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:55.308 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:55.308 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:55.323 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-10-20T03:35:55.365 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:55.364+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:55.442 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:55.442 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:55.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-10-20T03:35:55.504 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:55.504+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:55.567 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:55.564+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:55.592 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:55.592 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:55.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-10-20T03:35:55.668 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:55.665+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:55.676 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:35:55.768 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:55.765+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:55.773 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:55.773 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:55.787 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-10-20T03:35:55.834 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:55.835 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:35:55.855 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:55.855 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:35:55.869 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:55.868+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:55.875 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:55.876 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:35:55.898 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:55.898 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:35:55.909 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:55.909 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2023-10-20T03:35:55.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2023-10-20T03:35:55.910 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2023-10-20T03:35:55.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.910 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:55.910 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.329122+0000", 2023-10-20T03:35:55.910 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.577405543000001, 2023-10-20T03:35:55.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034738698999999998, 2023-10-20T03:35:55.911 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:55.911 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:55.911 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:55.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:55.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329122+0000", 2023-10-20T03:35:55.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:55.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329122+0000", 2023-10-20T03:35:55.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:55.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329122+0000", 2023-10-20T03:35:55.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6399999999999998e-07 2023-10-20T03:35:55.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:55.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329551+0000", 2023-10-20T03:35:55.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042853900000000002 2023-10-20T03:35:55.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:55.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329577+0000", 2023-10-20T03:35:55.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6075e-05 2023-10-20T03:35:55.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:55.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329594+0000", 2023-10-20T03:35:55.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7220000000000001e-05 2023-10-20T03:35:55.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:55.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329788+0000", 2023-10-20T03:35:55.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000193444 2023-10-20T03:35:55.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:55.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329815+0000", 2023-10-20T03:35:55.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7107e-05 2023-10-20T03:35:55.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:55.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.363772+0000", 2023-10-20T03:35:55.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033956919000000002 2023-10-20T03:35:55.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:55.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.363861+0000", 2023-10-20T03:35:55.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9030999999999995e-05 2023-10-20T03:35:55.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.924 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:55.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.924 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 43/31 ECSubWrite(tid=81, reqid=client.4570.0:307, at_version=43'15, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:55.925 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.342678+0000", 2023-10-20T03:35:55.925 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.563849461, 2023-10-20T03:35:55.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021268994999999999, 2023-10-20T03:35:55.925 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:55.925 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:55.926 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:55.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:55.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.342678+0000", 2023-10-20T03:35:55.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:55.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.342678+0000", 2023-10-20T03:35:55.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:55.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.342680+0000", 2023-10-20T03:35:55.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.736e-06 2023-10-20T03:35:55.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:55.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343038+0000", 2023-10-20T03:35:55.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035823799999999998 2023-10-20T03:35:55.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:55.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343066+0000", 2023-10-20T03:35:55.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7509e-05 2023-10-20T03:35:55.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:55.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343081+0000", 2023-10-20T03:35:55.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5676e-05 2023-10-20T03:35:55.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:55.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343490+0000", 2023-10-20T03:35:55.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040832599999999998 2023-10-20T03:35:55.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:55.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343517+0000", 2023-10-20T03:35:55.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7639000000000001e-05 2023-10-20T03:35:55.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:55.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.363880+0000", 2023-10-20T03:35:55.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020362392999999999 2023-10-20T03:35:55.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:55.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.363947+0000", 2023-10-20T03:35:55.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7477999999999993e-05 2023-10-20T03:35:55.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.937 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:55.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.938 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s3 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:35:55.938 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.357002+0000", 2023-10-20T03:35:55.938 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.549525717000002, 2023-10-20T03:35:55.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024440087999999999, 2023-10-20T03:35:55.939 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:55.939 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:55.939 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:55.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:55.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357002+0000", 2023-10-20T03:35:55.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:55.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357002+0000", 2023-10-20T03:35:55.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:55.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357004+0000", 2023-10-20T03:35:55.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.883e-06 2023-10-20T03:35:55.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:55.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357470+0000", 2023-10-20T03:35:55.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046657099999999999 2023-10-20T03:35:55.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:55.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357514+0000", 2023-10-20T03:35:55.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3288999999999998e-05 2023-10-20T03:35:55.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:55.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357533+0000", 2023-10-20T03:35:55.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9123e-05 2023-10-20T03:35:55.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:55.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357687+0000", 2023-10-20T03:35:55.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000154389 2023-10-20T03:35:55.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:55.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357703+0000", 2023-10-20T03:35:55.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6062999999999999e-05 2023-10-20T03:35:55.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:55.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.381375+0000", 2023-10-20T03:35:55.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023672364000000001 2023-10-20T03:35:55.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:55.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.381442+0000", 2023-10-20T03:35:55.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6406000000000001e-05 2023-10-20T03:35:55.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.949 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:55.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.950 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs4 53/47 ECSubWrite(tid=57, reqid=client.4570.0:408, at_version=53'31, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:55.950 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:35.448730+0000", 2023-10-20T03:35:55.950 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.457796994999999, 2023-10-20T03:35:55.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015562098999999999, 2023-10-20T03:35:55.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:55.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:55.951 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:55.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:55.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.448730+0000", 2023-10-20T03:35:55.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:55.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.448730+0000", 2023-10-20T03:35:55.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:55.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.448732+0000", 2023-10-20T03:35:55.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099999999999999e-06 2023-10-20T03:35:55.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:55.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449349+0000", 2023-10-20T03:35:55.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061657400000000005 2023-10-20T03:35:55.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:55.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449378+0000", 2023-10-20T03:35:55.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9374999999999999e-05 2023-10-20T03:35:55.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:55.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449401+0000", 2023-10-20T03:35:55.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.296e-05 2023-10-20T03:35:55.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:55.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449514+0000", 2023-10-20T03:35:55.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113174 2023-10-20T03:35:55.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:55.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449533+0000", 2023-10-20T03:35:55.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8717000000000001e-05 2023-10-20T03:35:55.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:55.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.464194+0000", 2023-10-20T03:35:55.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014661502 2023-10-20T03:35:55.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:55.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.464293+0000", 2023-10-20T03:35:55.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8287000000000003e-05 2023-10-20T03:35:55.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.961 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:55.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.962 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs4 53/47 ECSubWrite(tid=58, reqid=client.4570.0:409, at_version=53'32, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:55.962 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:35.472576+0000", 2023-10-20T03:35:55.963 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.433951202999999, 2023-10-20T03:35:55.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022673781, 2023-10-20T03:35:55.963 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:55.963 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:55.964 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:55.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:55.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472576+0000", 2023-10-20T03:35:55.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:55.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472576+0000", 2023-10-20T03:35:55.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:55.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472578+0000", 2023-10-20T03:35:55.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2023-10-20T03:35:55.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:55.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472994+0000", 2023-10-20T03:35:55.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041626100000000002 2023-10-20T03:35:55.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:55.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473024+0000", 2023-10-20T03:35:55.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9278999999999999e-05 2023-10-20T03:35:55.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:55.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473046+0000", 2023-10-20T03:35:55.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2353000000000001e-05 2023-10-20T03:35:55.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:55.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473185+0000", 2023-10-20T03:35:55.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013938399999999999 2023-10-20T03:35:55.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:55.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473201+0000", 2023-10-20T03:35:55.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5342999999999999e-05 2023-10-20T03:35:55.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:55.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.495162+0000", 2023-10-20T03:35:55.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021961633000000001 2023-10-20T03:35:55.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:55.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.495250+0000", 2023-10-20T03:35:55.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7791e-05 2023-10-20T03:35:55.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.974 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:55.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.975 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 54/31 ECSubWrite(tid=65, reqid=client.4570.0:418, at_version=54'32, trim_to=0'0, roll_forward_to=51'31))", 2023-10-20T03:35:55.975 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:36.458790+0000", 2023-10-20T03:35:55.975 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.447737702000001, 2023-10-20T03:35:55.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015152827000000001, 2023-10-20T03:35:55.975 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:55.975 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:55.976 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:55.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:55.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.458790+0000", 2023-10-20T03:35:55.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:55.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.458790+0000", 2023-10-20T03:35:55.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:55.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.458791+0000", 2023-10-20T03:35:55.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5200000000000001e-06 2023-10-20T03:35:55.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:55.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.459266+0000", 2023-10-20T03:35:55.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047446599999999999 2023-10-20T03:35:55.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:55.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.459297+0000", 2023-10-20T03:35:55.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0948999999999998e-05 2023-10-20T03:35:55.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:55.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.459312+0000", 2023-10-20T03:35:55.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5526e-05 2023-10-20T03:35:55.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:55.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.459439+0000", 2023-10-20T03:35:55.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012685800000000001 2023-10-20T03:35:55.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:55.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.459456+0000", 2023-10-20T03:35:55.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7289e-05 2023-10-20T03:35:55.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:55.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.473843+0000", 2023-10-20T03:35:55.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014386421 2023-10-20T03:35:55.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:55.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.473943+0000", 2023-10-20T03:35:55.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9797999999999996e-05 2023-10-20T03:35:55.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.986 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:55.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.987 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 60/31 ECSubWrite(tid=190, reqid=client.4570.0:452, at_version=60'54, trim_to=0'0, roll_forward_to=58'48))", 2023-10-20T03:35:55.987 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:41.856865+0000", 2023-10-20T03:35:55.987 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.049662695, 2023-10-20T03:35:55.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016682313000000001, 2023-10-20T03:35:55.988 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:55.988 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:55.988 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:55.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:55.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.856865+0000", 2023-10-20T03:35:55.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:55.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.856865+0000", 2023-10-20T03:35:55.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:55.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:55.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.856866+0000", 2023-10-20T03:35:55.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.09e-07 2023-10-20T03:35:55.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:55.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857372+0000", 2023-10-20T03:35:55.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050624299999999997 2023-10-20T03:35:55.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:55.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857398+0000", 2023-10-20T03:35:55.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5803e-05 2023-10-20T03:35:55.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:55.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857415+0000", 2023-10-20T03:35:55.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7546000000000001e-05 2023-10-20T03:35:55.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:55.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857572+0000", 2023-10-20T03:35:55.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015650700000000001 2023-10-20T03:35:55.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:55.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857601+0000", 2023-10-20T03:35:55.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9513e-05 2023-10-20T03:35:55.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:55.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.873347+0000", 2023-10-20T03:35:55.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015746229 2023-10-20T03:35:55.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:55.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.873547+0000", 2023-10-20T03:35:55.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019966299999999999 2023-10-20T03:35:55.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.997 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:55.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:55.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:55.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:55.998 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 60/31 ECSubWrite(tid=45, reqid=osd.7.0:45, at_version=60'32, trim_to=0'0, roll_forward_to=56'30))", 2023-10-20T03:35:55.998 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:41.857583+0000", 2023-10-20T03:35:55.999 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.048944901, 2023-10-20T03:35:55.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016145725999999999, 2023-10-20T03:35:55.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:55.999 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:55.999 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857583+0000", 2023-10-20T03:35:56.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857583+0000", 2023-10-20T03:35:56.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857583+0000", 2023-10-20T03:35:56.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0699999999999996e-07 2023-10-20T03:35:56.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857634+0000", 2023-10-20T03:35:56.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0785999999999998e-05 2023-10-20T03:35:56.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857660+0000", 2023-10-20T03:35:56.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6363000000000002e-05 2023-10-20T03:35:56.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857682+0000", 2023-10-20T03:35:56.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1559000000000001e-05 2023-10-20T03:35:56.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857833+0000", 2023-10-20T03:35:56.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015113 2023-10-20T03:35:56.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857849+0000", 2023-10-20T03:35:56.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5906999999999999e-05 2023-10-20T03:35:56.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.873555+0000", 2023-10-20T03:35:56.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015706210000000002 2023-10-20T03:35:56.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.873728+0000", 2023-10-20T03:35:56.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017306400000000001 2023-10-20T03:35:56.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.008 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.009 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 60/31 ECSubWrite(tid=46, reqid=osd.7.0:46, at_version=60'34, trim_to=0'0, roll_forward_to=56'30))", 2023-10-20T03:35:56.009 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:41.863005+0000", 2023-10-20T03:35:56.009 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.043522691, 2023-10-20T03:35:56.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034635154000000001, 2023-10-20T03:35:56.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.010 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.010 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.863005+0000", 2023-10-20T03:35:56.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.863005+0000", 2023-10-20T03:35:56.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.863006+0000", 2023-10-20T03:35:56.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2023-10-20T03:35:56.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.863036+0000", 2023-10-20T03:35:56.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0494e-05 2023-10-20T03:35:56.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.863048+0000", 2023-10-20T03:35:56.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1643e-05 2023-10-20T03:35:56.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.863056+0000", 2023-10-20T03:35:56.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7190000000000007e-06 2023-10-20T03:35:56.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.870966+0000", 2023-10-20T03:35:56.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079100249999999993 2023-10-20T03:35:56.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.870994+0000", 2023-10-20T03:35:56.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8007000000000001e-05 2023-10-20T03:35:56.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.897561+0000", 2023-10-20T03:35:56.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026566751999999999 2023-10-20T03:35:56.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.897640+0000", 2023-10-20T03:35:56.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8769000000000004e-05 2023-10-20T03:35:56.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.020 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.021 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 60/31 ECSubWrite(tid=195, reqid=client.4570.0:453, at_version=60'55, trim_to=0'0, roll_forward_to=60'52))", 2023-10-20T03:35:56.021 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:41.877092+0000", 2023-10-20T03:35:56.021 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.029435877999999, 2023-10-20T03:35:56.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028928008000000002, 2023-10-20T03:35:56.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.022 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.022 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877092+0000", 2023-10-20T03:35:56.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877092+0000", 2023-10-20T03:35:56.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877093+0000", 2023-10-20T03:35:56.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.657e-06 2023-10-20T03:35:56.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877617+0000", 2023-10-20T03:35:56.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052373199999999995 2023-10-20T03:35:56.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877655+0000", 2023-10-20T03:35:56.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8253000000000001e-05 2023-10-20T03:35:56.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877685+0000", 2023-10-20T03:35:56.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0096000000000001e-05 2023-10-20T03:35:56.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877884+0000", 2023-10-20T03:35:56.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000198686 2023-10-20T03:35:56.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877915+0000", 2023-10-20T03:35:56.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0809000000000003e-05 2023-10-20T03:35:56.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.905863+0000", 2023-10-20T03:35:56.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027948218 2023-10-20T03:35:56.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.906020+0000", 2023-10-20T03:35:56.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015655700000000001 2023-10-20T03:35:56.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.031 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.032 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 64/31 ECSubWrite(tid=246, reqid=client.4570.0:539, at_version=64'63, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:35:56.032 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.897808+0000", 2023-10-20T03:35:56.032 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.008719195999999, 2023-10-20T03:35:56.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031221839000000001, 2023-10-20T03:35:56.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.033 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.033 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.897808+0000", 2023-10-20T03:35:56.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.897808+0000", 2023-10-20T03:35:56.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.897809+0000", 2023-10-20T03:35:56.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5999999999999999e-07 2023-10-20T03:35:56.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898180+0000", 2023-10-20T03:35:56.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000371443 2023-10-20T03:35:56.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898207+0000", 2023-10-20T03:35:56.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7059e-05 2023-10-20T03:35:56.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898225+0000", 2023-10-20T03:35:56.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7416999999999999e-05 2023-10-20T03:35:56.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898433+0000", 2023-10-20T03:35:56.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020842899999999999 2023-10-20T03:35:56.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898469+0000", 2023-10-20T03:35:56.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5896000000000003e-05 2023-10-20T03:35:56.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.928921+0000", 2023-10-20T03:35:56.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030451733000000002 2023-10-20T03:35:56.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.929030+0000", 2023-10-20T03:35:56.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109502 2023-10-20T03:35:56.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.043 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.044 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 64/31 ECSubWrite(tid=247, reqid=client.4570.0:540, at_version=64'64, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:35:56.044 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.912251+0000", 2023-10-20T03:35:56.044 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9942767799999999, 2023-10-20T03:35:56.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029618076, 2023-10-20T03:35:56.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.045 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.045 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912251+0000", 2023-10-20T03:35:56.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912251+0000", 2023-10-20T03:35:56.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912252+0000", 2023-10-20T03:35:56.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7099999999999999e-06 2023-10-20T03:35:56.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912538+0000", 2023-10-20T03:35:56.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028583800000000001 2023-10-20T03:35:56.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912591+0000", 2023-10-20T03:35:56.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2695999999999998e-05 2023-10-20T03:35:56.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912612+0000", 2023-10-20T03:35:56.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2023-10-20T03:35:56.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.917025+0000", 2023-10-20T03:35:56.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044131070000000003 2023-10-20T03:35:56.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.917041+0000", 2023-10-20T03:35:56.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.63e-05 2023-10-20T03:35:56.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.941770+0000", 2023-10-20T03:35:56.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024729004999999998 2023-10-20T03:35:56.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.941869+0000", 2023-10-20T03:35:56.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8863999999999994e-05 2023-10-20T03:35:56.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.055 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.056 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 64/31 ECSubWrite(tid=248, reqid=client.4570.0:541, at_version=64'65, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:35:56.056 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.926506+0000", 2023-10-20T03:35:56.056 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9800211940000008, 2023-10-20T03:35:56.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016346448999999999, 2023-10-20T03:35:56.057 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.057 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.057 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926506+0000", 2023-10-20T03:35:56.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926506+0000", 2023-10-20T03:35:56.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926508+0000", 2023-10-20T03:35:56.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7239999999999999e-06 2023-10-20T03:35:56.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926740+0000", 2023-10-20T03:35:56.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023180499999999999 2023-10-20T03:35:56.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926772+0000", 2023-10-20T03:35:56.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1803e-05 2023-10-20T03:35:56.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926795+0000", 2023-10-20T03:35:56.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3105999999999999e-05 2023-10-20T03:35:56.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.929124+0000", 2023-10-20T03:35:56.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002329607 2023-10-20T03:35:56.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.929158+0000", 2023-10-20T03:35:56.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3949000000000003e-05 2023-10-20T03:35:56.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.942766+0000", 2023-10-20T03:35:56.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013607842 2023-10-20T03:35:56.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.942853+0000", 2023-10-20T03:35:56.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6613000000000006e-05 2023-10-20T03:35:56.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.068 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 69/31 ECSubWrite(tid=95, reqid=client.4570.0:552, at_version=69'41, trim_to=0'0, roll_forward_to=56'39))", 2023-10-20T03:35:56.068 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:50.953519+0000", 2023-10-20T03:35:56.068 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.9530088829999999, 2023-10-20T03:35:56.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032960562999999998, 2023-10-20T03:35:56.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.069 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.953519+0000", 2023-10-20T03:35:56.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.953519+0000", 2023-10-20T03:35:56.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.953519+0000", 2023-10-20T03:35:56.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7e-07 2023-10-20T03:35:56.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.954577+0000", 2023-10-20T03:35:56.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001058072 2023-10-20T03:35:56.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.954609+0000", 2023-10-20T03:35:56.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1637000000000002e-05 2023-10-20T03:35:56.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.954630+0000", 2023-10-20T03:35:56.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1719e-05 2023-10-20T03:35:56.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.954811+0000", 2023-10-20T03:35:56.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018069399999999999 2023-10-20T03:35:56.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.954842+0000", 2023-10-20T03:35:56.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1053000000000002e-05 2023-10-20T03:35:56.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.986384+0000", 2023-10-20T03:35:56.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031542002999999999 2023-10-20T03:35:56.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.986479+0000", 2023-10-20T03:35:56.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5014999999999997e-05 2023-10-20T03:35:56.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.078 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.079 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.3s5 70/56 ECSubWrite(tid=55, reqid=client.4570.0:556, at_version=70'40, trim_to=0'0, roll_forward_to=60'38))", 2023-10-20T03:35:56.079 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.966477+0000", 2023-10-20T03:35:56.080 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.940050491, 2023-10-20T03:35:56.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016354670000000002, 2023-10-20T03:35:56.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.080 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.080 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966477+0000", 2023-10-20T03:35:56.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966477+0000", 2023-10-20T03:35:56.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966495+0000", 2023-10-20T03:35:56.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7666e-05 2023-10-20T03:35:56.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966636+0000", 2023-10-20T03:35:56.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141755 2023-10-20T03:35:56.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966667+0000", 2023-10-20T03:35:56.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0846000000000002e-05 2023-10-20T03:35:56.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966684+0000", 2023-10-20T03:35:56.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7173e-05 2023-10-20T03:35:56.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967319+0000", 2023-10-20T03:35:56.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063411999999999995 2023-10-20T03:35:56.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967349+0000", 2023-10-20T03:35:56.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0295999999999999e-05 2023-10-20T03:35:56.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.982759+0000", 2023-10-20T03:35:56.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015409707999999999 2023-10-20T03:35:56.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.982832+0000", 2023-10-20T03:35:56.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3106000000000001e-05 2023-10-20T03:35:56.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.090 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.091 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.3s5 70/56 ECSubWrite(tid=57, reqid=client.4570.0:558, at_version=70'42, trim_to=0'0, roll_forward_to=70'40))", 2023-10-20T03:35:56.091 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.998679+0000", 2023-10-20T03:35:56.091 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9078484759999998, 2023-10-20T03:35:56.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013262535000000001, 2023-10-20T03:35:56.092 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.092 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.092 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.998679+0000", 2023-10-20T03:35:56.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.998679+0000", 2023-10-20T03:35:56.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.998680+0000", 2023-10-20T03:35:56.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2890000000000001e-06 2023-10-20T03:35:56.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.998841+0000", 2023-10-20T03:35:56.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016102600000000001 2023-10-20T03:35:56.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.998871+0000", 2023-10-20T03:35:56.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.934e-05 2023-10-20T03:35:56.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.998887+0000", 2023-10-20T03:35:56.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6475999999999999e-05 2023-10-20T03:35:56.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.999113+0000", 2023-10-20T03:35:56.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022557999999999999 2023-10-20T03:35:56.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.999136+0000", 2023-10-20T03:35:56.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2954000000000001e-05 2023-10-20T03:35:56.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.011843+0000", 2023-10-20T03:35:56.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012706832 2023-10-20T03:35:56.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.011942+0000", 2023-10-20T03:35:56.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9037999999999996e-05 2023-10-20T03:35:56.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.102 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.103 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 70/31 ECSubWrite(tid=48, reqid=client.4570.0:567, at_version=70'36, trim_to=0'0, roll_forward_to=60'34))", 2023-10-20T03:35:56.103 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.044845+0000", 2023-10-20T03:35:56.103 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8616825700000001, 2023-10-20T03:35:56.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018144609999999999, 2023-10-20T03:35:56.104 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.104 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.104 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.044845+0000", 2023-10-20T03:35:56.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.044845+0000", 2023-10-20T03:35:56.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.044846+0000", 2023-10-20T03:35:56.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3300000000000001e-07 2023-10-20T03:35:56.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.045358+0000", 2023-10-20T03:35:56.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00051205900000000004 2023-10-20T03:35:56.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.045396+0000", 2023-10-20T03:35:56.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8729000000000001e-05 2023-10-20T03:35:56.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.045422+0000", 2023-10-20T03:35:56.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5256e-05 2023-10-20T03:35:56.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.045580+0000", 2023-10-20T03:35:56.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015875899999999999 2023-10-20T03:35:56.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.045604+0000", 2023-10-20T03:35:56.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3941e-05 2023-10-20T03:35:56.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.062900+0000", 2023-10-20T03:35:56.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017295535000000001 2023-10-20T03:35:56.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.062990+0000", 2023-10-20T03:35:56.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9598000000000005e-05 2023-10-20T03:35:56.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.114 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.115 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 70/31 ECSubWrite(tid=49, reqid=client.4570.0:568, at_version=70'37, trim_to=0'0, roll_forward_to=60'34))", 2023-10-20T03:35:56.115 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.065413+0000", 2023-10-20T03:35:56.115 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.841114583, 2023-10-20T03:35:56.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015280958000000001, 2023-10-20T03:35:56.115 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.116 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.116 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065413+0000", 2023-10-20T03:35:56.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065413+0000", 2023-10-20T03:35:56.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065414+0000", 2023-10-20T03:35:56.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.198e-06 2023-10-20T03:35:56.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065685+0000", 2023-10-20T03:35:56.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000271153 2023-10-20T03:35:56.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065709+0000", 2023-10-20T03:35:56.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3431999999999999e-05 2023-10-20T03:35:56.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065727+0000", 2023-10-20T03:35:56.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8343999999999999e-05 2023-10-20T03:35:56.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065828+0000", 2023-10-20T03:35:56.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100807 2023-10-20T03:35:56.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065842+0000", 2023-10-20T03:35:56.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4205999999999999e-05 2023-10-20T03:35:56.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.080645+0000", 2023-10-20T03:35:56.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014803278 2023-10-20T03:35:56.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.080694+0000", 2023-10-20T03:35:56.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8539999999999999e-05 2023-10-20T03:35:56.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.125 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.126 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.3s5 71/56 ECSubWrite(tid=63, reqid=client.4570.0:575, at_version=71'46, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:35:56.126 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.983660+0000", 2023-10-20T03:35:56.126 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.9228671180000001, 2023-10-20T03:35:56.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022066749, 2023-10-20T03:35:56.127 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.127 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.127 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.983660+0000", 2023-10-20T03:35:56.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.983660+0000", 2023-10-20T03:35:56.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.983677+0000", 2023-10-20T03:35:56.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7067000000000001e-05 2023-10-20T03:35:56.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984054+0000", 2023-10-20T03:35:56.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037618299999999999 2023-10-20T03:35:56.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984157+0000", 2023-10-20T03:35:56.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103799 2023-10-20T03:35:56.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984180+0000", 2023-10-20T03:35:56.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2178999999999999e-05 2023-10-20T03:35:56.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984376+0000", 2023-10-20T03:35:56.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195937 2023-10-20T03:35:56.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984413+0000", 2023-10-20T03:35:56.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7076000000000002e-05 2023-10-20T03:35:56.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.005641+0000", 2023-10-20T03:35:56.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021228526000000001 2023-10-20T03:35:56.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.005727+0000", 2023-10-20T03:35:56.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5982000000000004e-05 2023-10-20T03:35:56.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.137 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.138 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.3s5 71/56 ECSubWrite(tid=64, reqid=client.4570.0:576, at_version=71'47, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:35:56.138 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:53.006396+0000", 2023-10-20T03:35:56.138 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.9001318290000002, 2023-10-20T03:35:56.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027149204999999999, 2023-10-20T03:35:56.138 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.139 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.139 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006396+0000", 2023-10-20T03:35:56.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006396+0000", 2023-10-20T03:35:56.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006398+0000", 2023-10-20T03:35:56.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1959999999999998e-06 2023-10-20T03:35:56.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006808+0000", 2023-10-20T03:35:56.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000409694 2023-10-20T03:35:56.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006863+0000", 2023-10-20T03:35:56.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5439000000000002e-05 2023-10-20T03:35:56.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006895+0000", 2023-10-20T03:35:56.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1715999999999999e-05 2023-10-20T03:35:56.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007082+0000", 2023-10-20T03:35:56.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018744399999999999 2023-10-20T03:35:56.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007118+0000", 2023-10-20T03:35:56.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5373000000000002e-05 2023-10-20T03:35:56.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.033480+0000", 2023-10-20T03:35:56.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026362421 2023-10-20T03:35:56.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.033545+0000", 2023-10-20T03:35:56.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4921999999999999e-05 2023-10-20T03:35:56.148 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.148 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.148 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.148 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.149 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.149 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:56.149 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-10-20T03:35:56.151 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:56.152 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:56.148+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:56.152 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:35:56.153 INFO:tasks.rados.rados.0.smithi170.stdout:252: done (0 left) 2023-10-20T03:35:56.153 INFO:tasks.rados.rados.0.smithi170.stdout:253: delete oid 34 current snap is 21 2023-10-20T03:35:56.153 INFO:tasks.rados.rados.0.smithi170.stdout:253: done (0 left) 2023-10-20T03:35:56.154 INFO:tasks.rados.rados.0.smithi170.stdout:254: rollback oid 1 current snap is 21 2023-10-20T03:35:56.154 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 1 to 18 2023-10-20T03:35:56.154 INFO:tasks.rados.rados.0.smithi170.stdout:255: copy_from oid 15 from oid 7 current snap is 21 2023-10-20T03:35:56.154 INFO:tasks.rados.rados.0.smithi170.stdout:256: setattr oid 39 current snap is 21 2023-10-20T03:35:56.155 INFO:tasks.rados.rados.0.smithi170.stdout:257: snap_remove snap 19 2023-10-20T03:35:56.155 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:56.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-10-20T03:35:56.156 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:35:56.156 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:56.157 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:56.157 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:35:56.157 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:56.158 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:35:56.158 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:56.159 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-10-20T03:35:56.159 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:56.160 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:35:56.160 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:56.160 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:35:56.161 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:56.161 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:35:56.162 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:56.162 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:35:56.163 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:56.163 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:35:56.163 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:56.164 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:56.165 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:35:56.165 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:56.251 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:56.248+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:56.329 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:56.329 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:56.330 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2023-10-20T03:35:56.330 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:56.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-10-20T03:35:56.388 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:56.388+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:56.452 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:56.449+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:56.467 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:56.468 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:56.468 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2023-10-20T03:35:56.468 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2023-10-20T03:35:56.468 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:56.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-10-20T03:35:56.553 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:56.549+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:56.654 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:56.655 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2023-10-20T03:35:56.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2023-10-20T03:35:56.655 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2023-10-20T03:35:56.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.655 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:56.656 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.330272+0000", 2023-10-20T03:35:56.656 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.321314248, 2023-10-20T03:35:56.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035047879999999997, 2023-10-20T03:35:56.656 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.656 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.657 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330272+0000", 2023-10-20T03:35:56.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330272+0000", 2023-10-20T03:35:56.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330273+0000", 2023-10-20T03:35:56.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6300000000000005e-07 2023-10-20T03:35:56.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330709+0000", 2023-10-20T03:35:56.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043642900000000001 2023-10-20T03:35:56.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330745+0000", 2023-10-20T03:35:56.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6161999999999997e-05 2023-10-20T03:35:56.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330770+0000", 2023-10-20T03:35:56.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5151999999999999e-05 2023-10-20T03:35:56.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.331008+0000", 2023-10-20T03:35:56.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023709800000000001 2023-10-20T03:35:56.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.331035+0000", 2023-10-20T03:35:56.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7178999999999998e-05 2023-10-20T03:35:56.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365221+0000", 2023-10-20T03:35:56.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034186753 2023-10-20T03:35:56.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365320+0000", 2023-10-20T03:35:56.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8543999999999997e-05 2023-10-20T03:35:56.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.667 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 43/31 ECSubWrite(tid=81, reqid=client.4570.0:307, at_version=43'15, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:56.667 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.343425+0000", 2023-10-20T03:35:56.667 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.308161699999999, 2023-10-20T03:35:56.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021973126999999999, 2023-10-20T03:35:56.668 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.668 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.668 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343425+0000", 2023-10-20T03:35:56.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343425+0000", 2023-10-20T03:35:56.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343427+0000", 2023-10-20T03:35:56.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3999999999999999e-06 2023-10-20T03:35:56.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343627+0000", 2023-10-20T03:35:56.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000200099 2023-10-20T03:35:56.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343664+0000", 2023-10-20T03:35:56.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7165000000000001e-05 2023-10-20T03:35:56.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343690+0000", 2023-10-20T03:35:56.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5732999999999999e-05 2023-10-20T03:35:56.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.345198+0000", 2023-10-20T03:35:56.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001507743 2023-10-20T03:35:56.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.345227+0000", 2023-10-20T03:35:56.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9546000000000001e-05 2023-10-20T03:35:56.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365338+0000", 2023-10-20T03:35:56.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020110209 2023-10-20T03:35:56.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365398+0000", 2023-10-20T03:35:56.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0232000000000002e-05 2023-10-20T03:35:56.677 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.678 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.678 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.678 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:423 3.3s0 3:c87f890b:::smithi17039300-50:head [append 0~786432] snapc f=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-10-20T03:35:56.679 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:36.491346+0000", 2023-10-20T03:35:56.679 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.160240032000001, 2023-10-20T03:35:56.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.014221971, 2023-10-20T03:35:56.679 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.679 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:56.679 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:56.679 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:56.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:56.680 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 423 2023-10-20T03:35:56.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.680 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.491346+0000", 2023-10-20T03:35:56.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.491346+0000", 2023-10-20T03:35:56.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.491349+0000", 2023-10-20T03:35:56.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7800000000000001e-06 2023-10-20T03:35:56.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.492722+0000", 2023-10-20T03:35:56.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013730470000000001 2023-10-20T03:35:56.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.492757+0000", 2023-10-20T03:35:56.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4798999999999997e-05 2023-10-20T03:35:56.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.492773+0000", 2023-10-20T03:35:56.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6192999999999999e-05 2023-10-20T03:35:56.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.492921+0000", 2023-10-20T03:35:56.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014757400000000001 2023-10-20T03:35:56.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2023-10-20T03:35:56.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.492926+0000", 2023-10-20T03:35:56.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5430000000000001e-06 2023-10-20T03:35:56.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.460373+0000", 2023-10-20T03:35:56.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.967446264 2023-10-20T03:35:56.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:56.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.461124+0000", 2023-10-20T03:35:56.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075168299999999995 2023-10-20T03:35:56.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.465469+0000", 2023-10-20T03:35:56.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043443919999999999 2023-10-20T03:35:56.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.485210+0000", 2023-10-20T03:35:56.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019740981000000001 2023-10-20T03:35:56.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:35:56.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.504890+0000", 2023-10-20T03:35:56.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019680842 2023-10-20T03:35:56.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.505568+0000", 2023-10-20T03:35:56.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067787299999999995 2023-10-20T03:35:56.693 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.693 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.693 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.694 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 64/31 ECSubWrite(tid=246, reqid=client.4570.0:539, at_version=64'63, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:35:56.694 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.898251+0000", 2023-10-20T03:35:56.694 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.753335606, 2023-10-20T03:35:56.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028526125999999999, 2023-10-20T03:35:56.695 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.695 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.695 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898251+0000", 2023-10-20T03:35:56.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898251+0000", 2023-10-20T03:35:56.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898251+0000", 2023-10-20T03:35:56.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2600000000000002e-07 2023-10-20T03:35:56.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898832+0000", 2023-10-20T03:35:56.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058099900000000001 2023-10-20T03:35:56.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898866+0000", 2023-10-20T03:35:56.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4060000000000003e-05 2023-10-20T03:35:56.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898894+0000", 2023-10-20T03:35:56.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7538999999999998e-05 2023-10-20T03:35:56.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899124+0000", 2023-10-20T03:35:56.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023028600000000001 2023-10-20T03:35:56.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899154+0000", 2023-10-20T03:35:56.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9485999999999999e-05 2023-10-20T03:35:56.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926698+0000", 2023-10-20T03:35:56.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027544361 2023-10-20T03:35:56.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926777+0000", 2023-10-20T03:35:56.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8769000000000004e-05 2023-10-20T03:35:56.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.706 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.706 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.706 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 64/31 ECSubWrite(tid=247, reqid=client.4570.0:540, at_version=64'64, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:35:56.707 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.912900+0000", 2023-10-20T03:35:56.707 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.738686856999999, 2023-10-20T03:35:56.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023310649999999999, 2023-10-20T03:35:56.707 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.707 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.707 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912900+0000", 2023-10-20T03:35:56.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912900+0000", 2023-10-20T03:35:56.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912902+0000", 2023-10-20T03:35:56.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4899999999999999e-06 2023-10-20T03:35:56.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913260+0000", 2023-10-20T03:35:56.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035820300000000003 2023-10-20T03:35:56.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913302+0000", 2023-10-20T03:35:56.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1556e-05 2023-10-20T03:35:56.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913325+0000", 2023-10-20T03:35:56.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2793e-05 2023-10-20T03:35:56.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.914626+0000", 2023-10-20T03:35:56.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013018789999999999 2023-10-20T03:35:56.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.914645+0000", 2023-10-20T03:35:56.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8399000000000001e-05 2023-10-20T03:35:56.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.936098+0000", 2023-10-20T03:35:56.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021452966 2023-10-20T03:35:56.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.936210+0000", 2023-10-20T03:35:56.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112364 2023-10-20T03:35:56.718 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.718 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.718 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.719 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:556 3.3s0 3:c0108eeb:::smithi17039300-21:head [append 0~638976] snapc 23=[23] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:35:56.719 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.956395+0000", 2023-10-20T03:35:56.719 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6951911710000003, 2023-10-20T03:35:56.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039551132000000003, 2023-10-20T03:35:56.719 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.720 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:56.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:56.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:56.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:56.720 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 556 2023-10-20T03:35:56.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.721 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.956395+0000", 2023-10-20T03:35:56.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.956395+0000", 2023-10-20T03:35:56.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.956398+0000", 2023-10-20T03:35:56.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.903e-06 2023-10-20T03:35:56.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.957575+0000", 2023-10-20T03:35:56.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001176912 2023-10-20T03:35:56.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.957610+0000", 2023-10-20T03:35:56.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4656000000000002e-05 2023-10-20T03:35:56.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.957626+0000", 2023-10-20T03:35:56.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6676000000000001e-05 2023-10-20T03:35:56.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.957768+0000", 2023-10-20T03:35:56.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141975 2023-10-20T03:35:56.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2023-10-20T03:35:56.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.957785+0000", 2023-10-20T03:35:56.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6612999999999998e-05 2023-10-20T03:35:56.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.963559+0000", 2023-10-20T03:35:56.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057735659999999999 2023-10-20T03:35:56.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:56.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.963781+0000", 2023-10-20T03:35:56.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022263 2023-10-20T03:35:56.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966469+0000", 2023-10-20T03:35:56.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026878179999999998 2023-10-20T03:35:56.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.983765+0000", 2023-10-20T03:35:56.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017296308 2023-10-20T03:35:56.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:35:56.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.995677+0000", 2023-10-20T03:35:56.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011912109000000001 2023-10-20T03:35:56.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.995946+0000", 2023-10-20T03:35:56.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026896599999999999 2023-10-20T03:35:56.734 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.734 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.735 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:557 3.3s0 3:c0108eeb:::smithi17039300-21:head [append 0~458752] snapc 23=[23] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:35:56.735 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.963864+0000", 2023-10-20T03:35:56.736 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6877227550000002, 2023-10-20T03:35:56.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050243923000000003, 2023-10-20T03:35:56.736 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.736 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:56.736 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:56.736 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:56.737 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:56.737 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 557 2023-10-20T03:35:56.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.737 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.963864+0000", 2023-10-20T03:35:56.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.963864+0000", 2023-10-20T03:35:56.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.963866+0000", 2023-10-20T03:35:56.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0169999999999999e-06 2023-10-20T03:35:56.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.964523+0000", 2023-10-20T03:35:56.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065709600000000003 2023-10-20T03:35:56.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.964548+0000", 2023-10-20T03:35:56.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5032999999999999e-05 2023-10-20T03:35:56.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.964588+0000", 2023-10-20T03:35:56.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9991999999999999e-05 2023-10-20T03:35:56.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.984454+0000", 2023-10-20T03:35:56.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019866051999999999 2023-10-20T03:35:56.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:56.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.984811+0000", 2023-10-20T03:35:56.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035713500000000002 2023-10-20T03:35:56.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.986904+0000", 2023-10-20T03:35:56.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002092867 2023-10-20T03:35:56.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.013182+0000", 2023-10-20T03:35:56.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026277805000000001 2023-10-20T03:35:56.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:35:56.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.013831+0000", 2023-10-20T03:35:56.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064906199999999995 2023-10-20T03:35:56.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.014108+0000", 2023-10-20T03:35:56.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027686399999999998 2023-10-20T03:35:56.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.749 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.750 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:558 3.3s0 3:c0108eeb:::smithi17039300-21:head [append 0~704512] snapc 23=[23] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:35:56.750 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.975672+0000", 2023-10-20T03:35:56.750 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.675914594, 2023-10-20T03:35:56.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039948122000000003, 2023-10-20T03:35:56.751 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.751 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:56.751 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:56.751 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:56.751 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:56.752 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 558 2023-10-20T03:35:56.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.752 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.975672+0000", 2023-10-20T03:35:56.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.975672+0000", 2023-10-20T03:35:56.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.975675+0000", 2023-10-20T03:35:56.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4999999999999999e-06 2023-10-20T03:35:56.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977096+0000", 2023-10-20T03:35:56.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014207480000000001 2023-10-20T03:35:56.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977137+0000", 2023-10-20T03:35:56.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1149000000000002e-05 2023-10-20T03:35:56.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977159+0000", 2023-10-20T03:35:56.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2269999999999999e-05 2023-10-20T03:35:56.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.996676+0000", 2023-10-20T03:35:56.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019517016000000002 2023-10-20T03:35:56.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:56.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.996904+0000", 2023-10-20T03:35:56.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022794699999999999 2023-10-20T03:35:56.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.998600+0000", 2023-10-20T03:35:56.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001695237 2023-10-20T03:35:56.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015338+0000", 2023-10-20T03:35:56.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016738376999999999 2023-10-20T03:35:56.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:35:56.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015447+0000", 2023-10-20T03:35:56.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010899 2023-10-20T03:35:56.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015620+0000", 2023-10-20T03:35:56.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172888 2023-10-20T03:35:56.764 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.764 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.764 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.765 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:559 3.3s0 3:c0108eeb:::smithi17039300-21:head [setxattr _header (57) in=7b] snapc 23=[23] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:35:56.765 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.977198+0000", 2023-10-20T03:35:56.765 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6743884180000004, 2023-10-20T03:35:56.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039461151999999999, 2023-10-20T03:35:56.766 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.766 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:56.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:56.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:56.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:56.767 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 559 2023-10-20T03:35:56.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.767 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977198+0000", 2023-10-20T03:35:56.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977198+0000", 2023-10-20T03:35:56.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977200+0000", 2023-10-20T03:35:56.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7600000000000001e-06 2023-10-20T03:35:56.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977225+0000", 2023-10-20T03:35:56.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4817e-05 2023-10-20T03:35:56.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977251+0000", 2023-10-20T03:35:56.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6139000000000002e-05 2023-10-20T03:35:56.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977268+0000", 2023-10-20T03:35:56.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7076e-05 2023-10-20T03:35:56.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.011294+0000", 2023-10-20T03:35:56.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034025979999999997 2023-10-20T03:35:56.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:56.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.011452+0000", 2023-10-20T03:35:56.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000158639 2023-10-20T03:35:56.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.012364+0000", 2023-10-20T03:35:56.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091161599999999997 2023-10-20T03:35:56.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015628+0000", 2023-10-20T03:35:56.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032641079999999999 2023-10-20T03:35:56.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:35:56.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015729+0000", 2023-10-20T03:35:56.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101335 2023-10-20T03:35:56.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.016659+0000", 2023-10-20T03:35:56.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00092968200000000003 2023-10-20T03:35:56.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.779 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.779 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.779 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:560 3.3s0 3:c0108eeb:::smithi17039300-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:35:56.780 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.977299+0000", 2023-10-20T03:35:56.780 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6742872660000003, 2023-10-20T03:35:56.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043501639000000002, 2023-10-20T03:35:56.780 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.780 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:35:56.780 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:56.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:56.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:56.781 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 560 2023-10-20T03:35:56.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.781 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977299+0000", 2023-10-20T03:35:56.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977299+0000", 2023-10-20T03:35:56.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977300+0000", 2023-10-20T03:35:56.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.373e-06 2023-10-20T03:35:56.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977307+0000", 2023-10-20T03:35:56.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.387e-06 2023-10-20T03:35:56.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977330+0000", 2023-10-20T03:35:56.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3359000000000001e-05 2023-10-20T03:35:56.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977346+0000", 2023-10-20T03:35:56.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6093999999999999e-05 2023-10-20T03:35:56.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015155+0000", 2023-10-20T03:35:56.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037808698000000002 2023-10-20T03:35:56.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:56.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015327+0000", 2023-10-20T03:35:56.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171554 2023-10-20T03:35:56.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.016792+0000", 2023-10-20T03:35:56.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014655930000000001 2023-10-20T03:35:56.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:56.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.017045+0000", 2023-10-20T03:35:56.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025238700000000002 2023-10-20T03:35:56.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.020801+0000", 2023-10-20T03:35:56.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003756194 2023-10-20T03:35:56.792 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.792 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.793 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.793 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:575 3.3s0 3:c87f890b:::smithi17039300-50:head [append 0~704512] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:35:56.793 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.979848+0000", 2023-10-20T03:35:56.794 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6717383560000001, 2023-10-20T03:35:56.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040944178999999997, 2023-10-20T03:35:56.794 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.794 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:56.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:56.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:56.795 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:56.795 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 575 2023-10-20T03:35:56.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.795 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.979848+0000", 2023-10-20T03:35:56.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.979848+0000", 2023-10-20T03:35:56.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.979850+0000", 2023-10-20T03:35:56.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2539999999999999e-06 2023-10-20T03:35:56.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.981043+0000", 2023-10-20T03:35:56.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011924699999999999 2023-10-20T03:35:56.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.981087+0000", 2023-10-20T03:35:56.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3968999999999999e-05 2023-10-20T03:35:56.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.981101+0000", 2023-10-20T03:35:56.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4722999999999999e-05 2023-10-20T03:35:56.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.981240+0000", 2023-10-20T03:35:56.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013845799999999999 2023-10-20T03:35:56.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:56.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.981460+0000", 2023-10-20T03:35:56.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022006599999999999 2023-10-20T03:35:56.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.983567+0000", 2023-10-20T03:35:56.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021068749999999998 2023-10-20T03:35:56.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.019744+0000", 2023-10-20T03:35:56.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036176940999999997 2023-10-20T03:35:56.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:35:56.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.020535+0000", 2023-10-20T03:35:56.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079162700000000004 2023-10-20T03:35:56.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.020792+0000", 2023-10-20T03:35:56.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000256796 2023-10-20T03:35:56.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.807 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.808 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:576 3.3s0 3:c87f890b:::smithi17039300-50:head [append 0~655360] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:35:56.808 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.984983+0000", 2023-10-20T03:35:56.808 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6666034549999997, 2023-10-20T03:35:56.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057738351, 2023-10-20T03:35:56.808 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.809 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:56.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:56.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:56.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:56.809 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 576 2023-10-20T03:35:56.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.810 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984983+0000", 2023-10-20T03:35:56.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984983+0000", 2023-10-20T03:35:56.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984986+0000", 2023-10-20T03:35:56.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7499999999999999e-06 2023-10-20T03:35:56.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.985845+0000", 2023-10-20T03:35:56.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085890599999999999 2023-10-20T03:35:56.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.985879+0000", 2023-10-20T03:35:56.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4016000000000001e-05 2023-10-20T03:35:56.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.985901+0000", 2023-10-20T03:35:56.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2163000000000001e-05 2023-10-20T03:35:56.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.002888+0000", 2023-10-20T03:35:56.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016987001000000002 2023-10-20T03:35:56.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:56.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.003224+0000", 2023-10-20T03:35:56.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000335908 2023-10-20T03:35:56.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006382+0000", 2023-10-20T03:35:56.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031583039999999998 2023-10-20T03:35:56.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.021138+0000", 2023-10-20T03:35:56.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014756438 2023-10-20T03:35:56.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:35:56.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.042493+0000", 2023-10-20T03:35:56.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021354610999999999 2023-10-20T03:35:56.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.042721+0000", 2023-10-20T03:35:56.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000228254 2023-10-20T03:35:56.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.822 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.822 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:577 3.3s0 3:c87f890b:::smithi17039300-50:head [append 0~507904] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:35:56.823 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.988803+0000", 2023-10-20T03:35:56.823 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.662783669, 2023-10-20T03:35:56.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054481475000000001, 2023-10-20T03:35:56.823 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.823 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:56.824 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:56.824 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:56.824 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:56.824 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 577 2023-10-20T03:35:56.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.825 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.988803+0000", 2023-10-20T03:35:56.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.988803+0000", 2023-10-20T03:35:56.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.988805+0000", 2023-10-20T03:35:56.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.793e-06 2023-10-20T03:35:56.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989567+0000", 2023-10-20T03:35:56.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076297099999999998 2023-10-20T03:35:56.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989595+0000", 2023-10-20T03:35:56.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7195999999999998e-05 2023-10-20T03:35:56.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989614+0000", 2023-10-20T03:35:56.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.931e-05 2023-10-20T03:35:56.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.021402+0000", 2023-10-20T03:35:56.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031787718999999999 2023-10-20T03:35:56.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:56.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.021783+0000", 2023-10-20T03:35:56.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000381648 2023-10-20T03:35:56.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.024185+0000", 2023-10-20T03:35:56.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002401612 2023-10-20T03:35:56.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.041211+0000", 2023-10-20T03:35:56.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017025795 2023-10-20T03:35:56.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:35:56.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.043097+0000", 2023-10-20T03:35:56.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018863269999999999 2023-10-20T03:35:56.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.043284+0000", 2023-10-20T03:35:56.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000187104 2023-10-20T03:35:56.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.836 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.837 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:578 3.3s0 3:c87f890b:::smithi17039300-50:head [setxattr _header (57) in=7b] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:35:56.837 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.989649+0000", 2023-10-20T03:35:56.837 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6619371159999998, 2023-10-20T03:35:56.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054676374999999999, 2023-10-20T03:35:56.838 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.838 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:56.838 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:56.838 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:56.839 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:56.839 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 578 2023-10-20T03:35:56.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.839 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989649+0000", 2023-10-20T03:35:56.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989649+0000", 2023-10-20T03:35:56.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989651+0000", 2023-10-20T03:35:56.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4559999999999999e-06 2023-10-20T03:35:56.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989662+0000", 2023-10-20T03:35:56.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1077000000000001e-05 2023-10-20T03:35:56.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989684+0000", 2023-10-20T03:35:56.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2189e-05 2023-10-20T03:35:56.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989701+0000", 2023-10-20T03:35:56.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6750000000000001e-05 2023-10-20T03:35:56.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.038393+0000", 2023-10-20T03:35:56.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048692566 2023-10-20T03:35:56.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:56.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.038741+0000", 2023-10-20T03:35:56.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000347315 2023-10-20T03:35:56.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.040161+0000", 2023-10-20T03:35:56.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014204409999999999 2023-10-20T03:35:56.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.041971+0000", 2023-10-20T03:35:56.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001809532 2023-10-20T03:35:56.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:35:56.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.044136+0000", 2023-10-20T03:35:56.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021652749999999999 2023-10-20T03:35:56.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.044326+0000", 2023-10-20T03:35:56.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000189774 2023-10-20T03:35:56.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.851 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.852 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:579 3.3s0 3:c87f890b:::smithi17039300-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:35:56.852 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.989733+0000", 2023-10-20T03:35:56.852 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.661853421, 2023-10-20T03:35:56.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058151636, 2023-10-20T03:35:56.852 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.853 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:35:56.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:56.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:56.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:56.853 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 579 2023-10-20T03:35:56.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.854 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989733+0000", 2023-10-20T03:35:56.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989733+0000", 2023-10-20T03:35:56.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989734+0000", 2023-10-20T03:35:56.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.406e-06 2023-10-20T03:35:56.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989742+0000", 2023-10-20T03:35:56.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3270000000000003e-06 2023-10-20T03:35:56.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989761+0000", 2023-10-20T03:35:56.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9326000000000001e-05 2023-10-20T03:35:56.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989778+0000", 2023-10-20T03:35:56.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.649e-05 2023-10-20T03:35:56.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.045369+0000", 2023-10-20T03:35:56.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055591030999999999 2023-10-20T03:35:56.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:56.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.045545+0000", 2023-10-20T03:35:56.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176732 2023-10-20T03:35:56.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.047885+0000", 2023-10-20T03:35:56.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023393239999999998 2023-10-20T03:35:56.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.863 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.864 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 71/56 ECSubWriteReply(tid=63, last_complete=71'46, committed=1, applied=1))", 2023-10-20T03:35:56.864 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.995273+0000", 2023-10-20T03:35:56.864 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6563136940000001, 2023-10-20T03:35:56.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024137965000000001, 2023-10-20T03:35:56.865 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.865 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.865 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995273+0000", 2023-10-20T03:35:56.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995273+0000", 2023-10-20T03:35:56.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995275+0000", 2023-10-20T03:35:56.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9740000000000001e-06 2023-10-20T03:35:56.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995287+0000", 2023-10-20T03:35:56.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1853e-05 2023-10-20T03:35:56.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995320+0000", 2023-10-20T03:35:56.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3973000000000002e-05 2023-10-20T03:35:56.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995339+0000", 2023-10-20T03:35:56.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8992999999999999e-05 2023-10-20T03:35:56.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.019333+0000", 2023-10-20T03:35:56.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023993596999999998 2023-10-20T03:35:56.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.019411+0000", 2023-10-20T03:35:56.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7575000000000002e-05 2023-10-20T03:35:56.872 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.872 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.873 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 71/56 ECSubWriteReply(tid=63, last_complete=71'46, committed=1, applied=1))", 2023-10-20T03:35:56.873 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.995405+0000", 2023-10-20T03:35:56.874 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6561817319999999, 2023-10-20T03:35:56.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024140920999999999, 2023-10-20T03:35:56.874 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.874 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.874 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995405+0000", 2023-10-20T03:35:56.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995405+0000", 2023-10-20T03:35:56.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995406+0000", 2023-10-20T03:35:56.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3599999999999999e-06 2023-10-20T03:35:56.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995416+0000", 2023-10-20T03:35:56.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9029999999999999e-06 2023-10-20T03:35:56.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995446+0000", 2023-10-20T03:35:56.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0358999999999999e-05 2023-10-20T03:35:56.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995458+0000", 2023-10-20T03:35:56.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1507e-05 2023-10-20T03:35:56.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.019489+0000", 2023-10-20T03:35:56.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024030969999999999 2023-10-20T03:35:56.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.019546+0000", 2023-10-20T03:35:56.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6821999999999999e-05 2023-10-20T03:35:56.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.882 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.883 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 71/56 ECSubWriteReply(tid=63, last_complete=71'46, committed=1, applied=1))", 2023-10-20T03:35:56.883 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.999951+0000", 2023-10-20T03:35:56.883 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.651635282, 2023-10-20T03:35:56.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019796978, 2023-10-20T03:35:56.884 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.884 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.884 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.999951+0000", 2023-10-20T03:35:56.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.999951+0000", 2023-10-20T03:35:56.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.999952+0000", 2023-10-20T03:35:56.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3740000000000001e-06 2023-10-20T03:35:56.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.999960+0000", 2023-10-20T03:35:56.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9759999999999995e-06 2023-10-20T03:35:56.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.999986+0000", 2023-10-20T03:35:56.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5072999999999999e-05 2023-10-20T03:35:56.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.999997+0000", 2023-10-20T03:35:56.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1643e-05 2023-10-20T03:35:56.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.019667+0000", 2023-10-20T03:35:56.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019670157000000001 2023-10-20T03:35:56.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.019748+0000", 2023-10-20T03:35:56.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0754999999999995e-05 2023-10-20T03:35:56.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.892 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.892 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 71/56 ECSubWriteReply(tid=63, last_complete=71'46, committed=1, applied=1))", 2023-10-20T03:35:56.893 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:53.000313+0000", 2023-10-20T03:35:56.893 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6512737579999999, 2023-10-20T03:35:56.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019627391000000001, 2023-10-20T03:35:56.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.894 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.000313+0000", 2023-10-20T03:35:56.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.000313+0000", 2023-10-20T03:35:56.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.000314+0000", 2023-10-20T03:35:56.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2300000000000001e-06 2023-10-20T03:35:56.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.000320+0000", 2023-10-20T03:35:56.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6000000000000003e-06 2023-10-20T03:35:56.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.000340+0000", 2023-10-20T03:35:56.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9596999999999999e-05 2023-10-20T03:35:56.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.000350+0000", 2023-10-20T03:35:56.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7629999999999995e-06 2023-10-20T03:35:56.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.019861+0000", 2023-10-20T03:35:56.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019511659000000001 2023-10-20T03:35:56.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.019940+0000", 2023-10-20T03:35:56.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8541999999999994e-05 2023-10-20T03:35:56.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.901 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.902 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.bs2 71/56 ECSubWrite(tid=267, reqid=osd.13.0:267, at_version=71'42, trim_to=0'0, roll_forward_to=70'40))", 2023-10-20T03:35:56.902 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:53.020609+0000", 2023-10-20T03:35:56.902 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6309771639999999, 2023-10-20T03:35:56.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020592599, 2023-10-20T03:35:56.903 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:56.903 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:56.903 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:56.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:56.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.020609+0000", 2023-10-20T03:35:56.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:56.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.020609+0000", 2023-10-20T03:35:56.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:56.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:56.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.020611+0000", 2023-10-20T03:35:56.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3260000000000001e-06 2023-10-20T03:35:56.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:56.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.020663+0000", 2023-10-20T03:35:56.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2766000000000003e-05 2023-10-20T03:35:56.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:56.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.020698+0000", 2023-10-20T03:35:56.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5049999999999998e-05 2023-10-20T03:35:56.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:56.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.020720+0000", 2023-10-20T03:35:56.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1529999999999999e-05 2023-10-20T03:35:56.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:56.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.021195+0000", 2023-10-20T03:35:56.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047510199999999998 2023-10-20T03:35:56.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:56.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.021209+0000", 2023-10-20T03:35:56.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3862999999999999e-05 2023-10-20T03:35:56.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:56.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.041082+0000", 2023-10-20T03:35:56.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019873227 2023-10-20T03:35:56.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:56.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:56.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:56.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.041202+0000", 2023-10-20T03:35:56.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119735 2023-10-20T03:35:56.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.913 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:56.914 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:56.914 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:56.914 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:56.652+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:56.915 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-10-20T03:35:56.917 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:56.917+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:57.000 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2023-10-20T03:35:57.009 INFO:tasks.rados.rados.0.smithi170.stdout:257: done (3 left) 2023-10-20T03:35:57.009 INFO:tasks.rados.rados.0.smithi170.stdout:258: read oid 44 snap -1 2023-10-20T03:35:57.010 INFO:tasks.rados.rados.0.smithi170.stdout:258: expect (ObjNum 43 snap 0 seq_num 43) 2023-10-20T03:35:57.010 INFO:tasks.rados.rados.0.smithi170.stdout:254: finishing rollback tid 0 to smithi17039300-1 2023-10-20T03:35:57.010 INFO:tasks.rados.rados.0.smithi170.stdout:254: finishing rollback tid 1 to smithi17039300-1 2023-10-20T03:35:57.010 INFO:tasks.rados.rados.0.smithi170.stdout:254: finishing rollback tid 2 to smithi17039300-1 2023-10-20T03:35:57.011 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 1 v 34 (ObjNum 77 snap 5 seq_num 77) dirty exists 2023-10-20T03:35:57.011 INFO:tasks.rados.rados.0.smithi170.stdout:255: finishing copy_from racing read to smithi17039300-15 2023-10-20T03:35:57.011 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 46 (ObjNum 2081 snap 0 seq_num 3160655632) dirty exists 2023-10-20T03:35:57.011 INFO:tasks.rados.rados.0.smithi170.stdout:255: finishing copy_from to smithi17039300-15 2023-10-20T03:35:57.012 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 15 v 19 (ObjNum 66 snap 4 seq_num 66) dirty exists 2023-10-20T03:35:57.012 INFO:tasks.rados.rados.0.smithi170.stdout:259: read oid 3 snap -1 2023-10-20T03:35:57.012 INFO:tasks.rados.rados.0.smithi170.stdout:259: expect (ObjNum 2 snap 0 seq_num 2) 2023-10-20T03:35:57.012 INFO:tasks.rados.rados.0.smithi170.stdout:254: done (4 left) 2023-10-20T03:35:57.012 INFO:tasks.rados.rados.0.smithi170.stdout:255: done (3 left) 2023-10-20T03:35:57.013 INFO:tasks.rados.rados.0.smithi170.stdout:256: done (2 left) 2023-10-20T03:35:57.013 INFO:tasks.rados.rados.0.smithi170.stdout:260: read oid 50 snap -1 2023-10-20T03:35:57.013 INFO:tasks.rados.rados.0.smithi170.stdout:260: expect (ObjNum 111 snap 18 seq_num 111) 2023-10-20T03:35:57.013 INFO:tasks.rados.rados.0.smithi170.stdout:261: copy_from oid 46 from oid 9 current snap is 21 2023-10-20T03:35:57.013 INFO:tasks.rados.rados.0.smithi170.stdout:262: snap_remove snap 20 2023-10-20T03:35:57.019 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:57.016+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:57.105 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:57.105 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:57.105 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2023-10-20T03:35:57.106 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:57.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-10-20T03:35:57.120 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:35:57.116+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:35:57.219 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:57.216+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:57.290 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:35:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2023-10-20T03:35:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2023-10-20T03:35:57.291 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:57.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-10-20T03:35:57.352 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:57.352+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:57.422 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:35:57.420+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:35:57.426 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:35:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2023-10-20T03:35:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2023-10-20T03:35:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2023-10-20T03:35:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.427 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es1 37/31 ECSubWrite(tid=23, reqid=client.4570.0:208, at_version=37'14, trim_to=0'0, roll_forward_to=36'12))", 2023-10-20T03:35:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:19.273182+0000", 2023-10-20T03:35:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.150431406000003, 2023-10-20T03:35:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016185486999999998, 2023-10-20T03:35:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.428 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.273182+0000", 2023-10-20T03:35:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.273182+0000", 2023-10-20T03:35:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.273182+0000", 2023-10-20T03:35:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5999999999999999e-07 2023-10-20T03:35:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.273913+0000", 2023-10-20T03:35:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073112000000000003 2023-10-20T03:35:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.273946+0000", 2023-10-20T03:35:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2792e-05 2023-10-20T03:35:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.273967+0000", 2023-10-20T03:35:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1279000000000001e-05 2023-10-20T03:35:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.274135+0000", 2023-10-20T03:35:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000167412 2023-10-20T03:35:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.274154+0000", 2023-10-20T03:35:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8872000000000002e-05 2023-10-20T03:35:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.289287+0000", 2023-10-20T03:35:57.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015133604 2023-10-20T03:35:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:19.289367+0000", 2023-10-20T03:35:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0048000000000003e-05 2023-10-20T03:35:57.438 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.439 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.329666+0000", 2023-10-20T03:35:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.093946666999997, 2023-10-20T03:35:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026825213000000001, 2023-10-20T03:35:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.440 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329666+0000", 2023-10-20T03:35:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329666+0000", 2023-10-20T03:35:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329667+0000", 2023-10-20T03:35:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2023-10-20T03:35:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330048+0000", 2023-10-20T03:35:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038081299999999997 2023-10-20T03:35:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330082+0000", 2023-10-20T03:35:57.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4402e-05 2023-10-20T03:35:57.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330100+0000", 2023-10-20T03:35:57.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7683e-05 2023-10-20T03:35:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330268+0000", 2023-10-20T03:35:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016784499999999999 2023-10-20T03:35:57.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330296+0000", 2023-10-20T03:35:57.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8367000000000001e-05 2023-10-20T03:35:57.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356426+0000", 2023-10-20T03:35:57.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026129432000000001 2023-10-20T03:35:57.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356492+0000", 2023-10-20T03:35:57.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6063999999999996e-05 2023-10-20T03:35:57.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.450 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.451 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 43/31 ECSubWrite(tid=81, reqid=client.4570.0:307, at_version=43'15, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:35:57.451 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.343216+0000", 2023-10-20T03:35:57.451 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.080396778999997, 2023-10-20T03:35:57.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016101977, 2023-10-20T03:35:57.451 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.452 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.452 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343216+0000", 2023-10-20T03:35:57.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343216+0000", 2023-10-20T03:35:57.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343219+0000", 2023-10-20T03:35:57.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1739999999999999e-06 2023-10-20T03:35:57.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343585+0000", 2023-10-20T03:35:57.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036599000000000001 2023-10-20T03:35:57.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343611+0000", 2023-10-20T03:35:57.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6749000000000001e-05 2023-10-20T03:35:57.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343627+0000", 2023-10-20T03:35:57.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5282999999999999e-05 2023-10-20T03:35:57.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.344698+0000", 2023-10-20T03:35:57.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010717910000000001 2023-10-20T03:35:57.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.344728+0000", 2023-10-20T03:35:57.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9332e-05 2023-10-20T03:35:57.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359214+0000", 2023-10-20T03:35:57.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014485924000000001 2023-10-20T03:35:57.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.359318+0000", 2023-10-20T03:35:57.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104734 2023-10-20T03:35:57.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.463 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.463 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s1 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:35:57.464 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.357167+0000", 2023-10-20T03:35:57.464 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.066445782000002, 2023-10-20T03:35:57.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025761703, 2023-10-20T03:35:57.464 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.464 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.465 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357167+0000", 2023-10-20T03:35:57.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357167+0000", 2023-10-20T03:35:57.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357169+0000", 2023-10-20T03:35:57.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.967e-06 2023-10-20T03:35:57.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357532+0000", 2023-10-20T03:35:57.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036283700000000002 2023-10-20T03:35:57.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357564+0000", 2023-10-20T03:35:57.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1798999999999998e-05 2023-10-20T03:35:57.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357615+0000", 2023-10-20T03:35:57.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1004999999999997e-05 2023-10-20T03:35:57.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357832+0000", 2023-10-20T03:35:57.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000216608 2023-10-20T03:35:57.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357857+0000", 2023-10-20T03:35:57.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5092999999999998e-05 2023-10-20T03:35:57.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.382875+0000", 2023-10-20T03:35:57.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025017978999999999 2023-10-20T03:35:57.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.382929+0000", 2023-10-20T03:35:57.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4415e-05 2023-10-20T03:35:57.474 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.475 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.475 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:320 3.2s0 3:474cd252:::smithi17039300-42:head [create,setxattr c[X]j^UsOa`q]sYrSaa]raqNm]iSd^X]OmelObeN (242) in=40b,setxattr gopkVbealbs^Z^^`^sXjNlPPYp^\\fbk\\fUnmTq[R (208) in=40b,setxattr eUZoNnss (4560) in=8b,setxattr SYo`knsRbgPQPPbXW[[ZsTehoXbSPY_WQl]_URc` (105) in=40b,setxattr \\PYbW]PkbTb^sYhsihYgqj]pROo]\\NWqNhT_lWgn (17753) in=40b,setxattr ]R_fObdq\\NUZ]QiO]j`djkVVp`bfRdph]Zbea]gg (17332) in=40b,setxattr ]ptcajatsVQWlWeoNNlPngWealknZSiQarONqbaV (1473) in=40b,setxattr `i_qfTQWSto[gYWTmYsOtOhsW[fnhaVacqaadrU] (1191) in=40b,setxattr nNhpa\\e^VfUjte_Pc[kSor^[gpr[eaflXRs]SWZr (16140) in=40b,setxattr aRNZ_mZpadm^l[rZntn`UX^RN\\cZ_TT]rRTjibjc (1121) in=40b,setxattr aVipjRkr_^aolT[aa`UQhsfb\\eRQ[ZZleb_OQOlX (2384) in=40b,setxattr m^VcgrkhPRmZlXiSggOaoPTi^]^WnQh\\aXaekmoc (947) in=40b,setxattr NikdcN_ilNkWgOQgZqrYPT\\b[drPRq\\niPPQdbVU (5260) in=40b,setxattr ajnWlVTfP^PVRs[VchU^kT_gSmcjmZRh]ofVgTrp (1437) in=40b,setxattr liRPRT`aa[cqbZkpaWa\\q[NZWUXYk[a[cicigTsV (1017) in=40b,setxattr XkR_Nkkpdeiacga[]Y^NismNTd`tQP[eb[qbrrd^ (8912) in=40b,setxattr ]mai`f_WjqaYertRarRZ[mrmWf_`mktRW`slld^_ (243) in=40b,setxattr jacQPeR\\bNj_nOTOc^WkPrSpceTds[[eee`SndWY (1086) in=40b,setxattr \\lcQhm[Vm\\jW\\cPdSlWoZcU[d^htQTrhRak`t\\hk (4068) in=40b,setxattr e`Od\\\\V]jhaQVknTVdWZOTq[eYbmlehRf^ZtoZnQ (384) in=40b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:57.476 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.333521+0000", 2023-10-20T03:35:57.476 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.090091649000001, 2023-10-20T03:35:57.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016194731, 2023-10-20T03:35:57.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:57.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:57.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:57.477 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:57.477 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 320 2023-10-20T03:35:57.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.478 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333521+0000", 2023-10-20T03:35:57.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333521+0000", 2023-10-20T03:35:57.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333524+0000", 2023-10-20T03:35:57.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8270000000000002e-06 2023-10-20T03:35:57.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333727+0000", 2023-10-20T03:35:57.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000202504 2023-10-20T03:35:57.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333762+0000", 2023-10-20T03:35:57.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5271999999999997e-05 2023-10-20T03:35:57.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333780+0000", 2023-10-20T03:35:57.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8280000000000001e-05 2023-10-20T03:35:57.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333968+0000", 2023-10-20T03:35:57.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000187655 2023-10-20T03:35:57.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2023-10-20T03:35:57.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333984+0000", 2023-10-20T03:35:57.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5739e-05 2023-10-20T03:35:57.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.339320+0000", 2023-10-20T03:35:57.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053365089999999997 2023-10-20T03:35:57.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:57.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.339685+0000", 2023-10-20T03:35:57.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036516599999999999 2023-10-20T03:35:57.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.342206+0000", 2023-10-20T03:35:57.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025201440000000002 2023-10-20T03:35:57.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.346469+0000", 2023-10-20T03:35:57.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042632420000000004 2023-10-20T03:35:57.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:35:57.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.349492+0000", 2023-10-20T03:35:57.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030232560000000002 2023-10-20T03:35:57.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.349716+0000", 2023-10-20T03:35:57.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022413700000000001 2023-10-20T03:35:57.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.491 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.492 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.492 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:387 3.2s0 3:4f9c85d4:::smithi17039300-39:d [read 0~1622016,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:35:57.492 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.381906+0000", 2023-10-20T03:35:57.493 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.041707109000001, 2023-10-20T03:35:57.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016689273000000001, 2023-10-20T03:35:57.493 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.493 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:35:57.493 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:57.493 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:57.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:57.494 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 387 2023-10-20T03:35:57.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.494 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381906+0000", 2023-10-20T03:35:57.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381906+0000", 2023-10-20T03:35:57.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381909+0000", 2023-10-20T03:35:57.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.965e-06 2023-10-20T03:35:57.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381925+0000", 2023-10-20T03:35:57.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5914999999999999e-05 2023-10-20T03:35:57.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381953+0000", 2023-10-20T03:35:57.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8430999999999999e-05 2023-10-20T03:35:57.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381968+0000", 2023-10-20T03:35:57.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4430000000000001e-05 2023-10-20T03:35:57.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382204+0000", 2023-10-20T03:35:57.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023644700000000001 2023-10-20T03:35:57.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2023-10-20T03:35:57.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382224+0000", 2023-10-20T03:35:57.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9406e-05 2023-10-20T03:35:57.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.387100+0000", 2023-10-20T03:35:57.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048765429999999997 2023-10-20T03:35:57.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:57.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.387279+0000", 2023-10-20T03:35:57.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017912599999999999 2023-10-20T03:35:57.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.398595+0000", 2023-10-20T03:35:57.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01131601 2023-10-20T03:35:57.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.506 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.507 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:388 3.2s0 3:4f9c85d4:::smithi17039300-39:d [read 0~1622016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:35:57.507 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.382005+0000", 2023-10-20T03:35:57.507 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.041608490000002, 2023-10-20T03:35:57.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019238755, 2023-10-20T03:35:57.508 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.508 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:35:57.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:57.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:57.508 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:57.509 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 388 2023-10-20T03:35:57.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.509 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382005+0000", 2023-10-20T03:35:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382005+0000", 2023-10-20T03:35:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382005+0000", 2023-10-20T03:35:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4200000000000005e-07 2023-10-20T03:35:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382011+0000", 2023-10-20T03:35:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0839999999999999e-06 2023-10-20T03:35:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382020+0000", 2023-10-20T03:35:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8709999999999992e-06 2023-10-20T03:35:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382029+0000", 2023-10-20T03:35:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0450000000000006e-06 2023-10-20T03:35:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382364+0000", 2023-10-20T03:35:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000334493 2023-10-20T03:35:57.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382374+0000", 2023-10-20T03:35:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.028e-05 2023-10-20T03:35:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.388188+0000", 2023-10-20T03:35:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058138490000000003 2023-10-20T03:35:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.388305+0000", 2023-10-20T03:35:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116747 2023-10-20T03:35:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.401243+0000", 2023-10-20T03:35:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012938544 2023-10-20T03:35:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.520 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:389 3.2s0 3:4f9c85d4:::smithi17039300-39:d [read 0~1622016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:35:57.521 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.382042+0000", 2023-10-20T03:35:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.041571529999999, 2023-10-20T03:35:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021763228999999999, 2023-10-20T03:35:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:35:57.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 389 2023-10-20T03:35:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.523 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382042+0000", 2023-10-20T03:35:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382042+0000", 2023-10-20T03:35:57.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382042+0000", 2023-10-20T03:35:57.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2023-10-20T03:35:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382046+0000", 2023-10-20T03:35:57.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0319999999999997e-06 2023-10-20T03:35:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382061+0000", 2023-10-20T03:35:57.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4800999999999999e-05 2023-10-20T03:35:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382067+0000", 2023-10-20T03:35:57.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0630000000000002e-06 2023-10-20T03:35:57.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382430+0000", 2023-10-20T03:35:57.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000362505 2023-10-20T03:35:57.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:57.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382439+0000", 2023-10-20T03:35:57.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2199999999999998e-06 2023-10-20T03:35:57.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.388649+0000", 2023-10-20T03:35:57.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0062098459999999998 2023-10-20T03:35:57.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:57.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.388756+0000", 2023-10-20T03:35:57.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107316 2023-10-20T03:35:57.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.403805+0000", 2023-10-20T03:35:57.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015048857000000001 2023-10-20T03:35:57.537 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.537 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.537 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.538 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs3 53/47 ECSubWrite(tid=57, reqid=client.4570.0:408, at_version=53'31, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:57.538 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:35.448862+0000", 2023-10-20T03:35:57.538 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.974751168000001, 2023-10-20T03:35:57.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015397816, 2023-10-20T03:35:57.538 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.539 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.539 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.448862+0000", 2023-10-20T03:35:57.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.448862+0000", 2023-10-20T03:35:57.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.448863+0000", 2023-10-20T03:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5e-06 2023-10-20T03:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449495+0000", 2023-10-20T03:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063173000000000005 2023-10-20T03:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449527+0000", 2023-10-20T03:35:57.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2066000000000001e-05 2023-10-20T03:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449542+0000", 2023-10-20T03:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5119999999999999e-05 2023-10-20T03:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449674+0000", 2023-10-20T03:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013180700000000001 2023-10-20T03:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449690+0000", 2023-10-20T03:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5316e-05 2023-10-20T03:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.464182+0000", 2023-10-20T03:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014492191999999999 2023-10-20T03:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.464260+0000", 2023-10-20T03:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8084999999999995e-05 2023-10-20T03:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.549 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs3 53/47 ECSubWrite(tid=58, reqid=client.4570.0:409, at_version=53'32, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:35.472767+0000", 2023-10-20T03:35:57.550 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.950846247000001, 2023-10-20T03:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022685556999999999, 2023-10-20T03:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472767+0000", 2023-10-20T03:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472767+0000", 2023-10-20T03:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472769+0000", 2023-10-20T03:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6869999999999999e-06 2023-10-20T03:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473140+0000", 2023-10-20T03:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037175199999999999 2023-10-20T03:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473171+0000", 2023-10-20T03:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0855000000000002e-05 2023-10-20T03:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473186+0000", 2023-10-20T03:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4946999999999999e-05 2023-10-20T03:35:57.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473343+0000", 2023-10-20T03:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015689 2023-10-20T03:35:57.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473369+0000", 2023-10-20T03:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6389000000000001e-05 2023-10-20T03:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.495356+0000", 2023-10-20T03:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021986362999999998 2023-10-20T03:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.495452+0000", 2023-10-20T03:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6674000000000003e-05 2023-10-20T03:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:433 3.2s0 3:474cd252:::smithi17039300-42:head [create,setxattr [VYQXaeer[Nsq`eTbgjNZfdhqbTRn_][lloSeZUP (16840) in=40b,setxattr hpaXc^]qjrqU_ciY[cWncm]SPa^tP_jRrRXm`jol (175) in=40b,setxattr YVpShpT[fhfamkUsgr]mb\\foasmX^k`jdhbbdlfP (1823) in=40b,setxattr ZhfkQl`OllQo^hemUnadbmqWdoabfgjcc^sVm_gf (1611) in=40b,setxattr aakOOaWemYRPYcaZ\\knQb (1051) in=21b,setxattr gPaagTUadWkOtqaRt]^rdUj^Zinmt]U^ (1051) in=32b,setxattr SfXmSZ]crX[dqcaqjNrrroQQbptW^Q^nUSUlcgdk (807) in=40b,setxattr SmRaehVPpUNpfZ[UX\\ZO_Ygfofk^ah^Xki^XTeX` (1191) in=40b,setxattr aTgW^YtXrjnrhT (1051) in=14b,setxattr idO[Vrlsc^pW[hYfh`UQSs^m_mU[YefWeld[YZ[c (1543) in=40b,setxattr ]Q``task\\rr]qaY[Pg`i[alh[rPlSdrcQ^iW^Z^g (1157) in=40b,setxattr _OOWpoQQg\\jtQUi``q_s[ae_akq`YOYilNTq^SPO (1121) in=40b,setxattr kN\\_ZaOX]q[kXen^Z]liYfaVP]prk\\r\\\\_Wsd[d_ (105) in=40b,setxattr OU`jdr^jTir[joOegmm (1051) in=19b,setxattr tRofbjROgtWqoRfmUVZXhpaZ[Q^sUNZb`\\YsdUfh (1507) in=40b,setxattr RRNpbt[`okmbR]RpSWfmmedXrUpaltcbWPbNRZRZ (67) in=40b,setxattr i[`^r^aojXRaU[[T\\kgWN^ (1051) in=22b,setxattr gZhSajUgdm`o]W[X`sgdVl^TtY^_SX[_l[Tr_aaO (663) in=40b,setxattr qN][QajljeddtQTcQX_QaTrg\\V\\pY\\QbsWehde`s (1646) in=40b,setxattr fbTrNt\\`ZVPVN[]tcg]iPXYOnmYnV]oaW_^o^gdr (1403) in=40b] snapc 16=[16,3,2] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2023-10-20T03:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:37.441170+0000", 2023-10-20T03:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.982443316000001, 2023-10-20T03:35:57.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020595808, 2023-10-20T03:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:35:57.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 433 2023-10-20T03:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441170+0000", 2023-10-20T03:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441170+0000", 2023-10-20T03:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441172+0000", 2023-10-20T03:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.469e-06 2023-10-20T03:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441400+0000", 2023-10-20T03:35:57.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000227226 2023-10-20T03:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441441+0000", 2023-10-20T03:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1644999999999999e-05 2023-10-20T03:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441467+0000", 2023-10-20T03:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5412999999999999e-05 2023-10-20T03:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441665+0000", 2023-10-20T03:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019792900000000001 2023-10-20T03:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441999+0000", 2023-10-20T03:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000334073 2023-10-20T03:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.445086+0000", 2023-10-20T03:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030878160000000002 2023-10-20T03:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.460222+0000", 2023-10-20T03:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015135984 2023-10-20T03:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.460491+0000", 2023-10-20T03:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026856699999999999 2023-10-20T03:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.461766+0000", 2023-10-20T03:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012746859999999999 2023-10-20T03:35:57.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.578 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs3 55/47 ECSubWrite(tid=71, reqid=client.4570.0:406, at_version=55'38, trim_to=0'0, roll_forward_to=54'36))", 2023-10-20T03:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:37.472200+0000", 2023-10-20T03:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.951413359, 2023-10-20T03:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02292278, 2023-10-20T03:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.579 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.472200+0000", 2023-10-20T03:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.472200+0000", 2023-10-20T03:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.472201+0000", 2023-10-20T03:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3230000000000001e-06 2023-10-20T03:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.473418+0000", 2023-10-20T03:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012171720000000001 2023-10-20T03:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.473456+0000", 2023-10-20T03:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8139000000000002e-05 2023-10-20T03:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.473479+0000", 2023-10-20T03:35:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2446000000000002e-05 2023-10-20T03:35:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.473634+0000", 2023-10-20T03:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015524000000000001 2023-10-20T03:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.473657+0000", 2023-10-20T03:35:57.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3190999999999999e-05 2023-10-20T03:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.495030+0000", 2023-10-20T03:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021373098 2023-10-20T03:35:57.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.495123+0000", 2023-10-20T03:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2170999999999996e-05 2023-10-20T03:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.588 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/31 ECSubWrite(tid=246, reqid=client.4570.0:539, at_version=64'63, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:35:57.589 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.898724+0000", 2023-10-20T03:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.524888860000001, 2023-10-20T03:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030365014999999999, 2023-10-20T03:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898724+0000", 2023-10-20T03:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898724+0000", 2023-10-20T03:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898725+0000", 2023-10-20T03:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2023-10-20T03:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899207+0000", 2023-10-20T03:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048263100000000002 2023-10-20T03:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899243+0000", 2023-10-20T03:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5139999999999999e-05 2023-10-20T03:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899266+0000", 2023-10-20T03:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3142999999999998e-05 2023-10-20T03:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899450+0000", 2023-10-20T03:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018391999999999999 2023-10-20T03:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899477+0000", 2023-10-20T03:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7889e-05 2023-10-20T03:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.928985+0000", 2023-10-20T03:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029507230999999998 2023-10-20T03:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.929089+0000", 2023-10-20T03:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104615 2023-10-20T03:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/31 ECSubWrite(tid=247, reqid=client.4570.0:540, at_version=64'64, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.913105+0000", 2023-10-20T03:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.510508508999999, 2023-10-20T03:35:57.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028811000999999999, 2023-10-20T03:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913105+0000", 2023-10-20T03:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913105+0000", 2023-10-20T03:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913106+0000", 2023-10-20T03:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6530000000000001e-06 2023-10-20T03:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913264+0000", 2023-10-20T03:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015735400000000001 2023-10-20T03:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913297+0000", 2023-10-20T03:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3099000000000003e-05 2023-10-20T03:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913312+0000", 2023-10-20T03:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5713e-05 2023-10-20T03:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.916852+0000", 2023-10-20T03:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003540007 2023-10-20T03:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.916885+0000", 2023-10-20T03:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2716000000000003e-05 2023-10-20T03:35:57.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.941775+0000", 2023-10-20T03:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024890223999999999 2023-10-20T03:35:57.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.941916+0000", 2023-10-20T03:35:57.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140235 2023-10-20T03:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/31 ECSubWrite(tid=248, reqid=client.4570.0:541, at_version=64'65, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.927074+0000", 2023-10-20T03:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.496539411000001, 2023-10-20T03:35:57.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015978672999999999, 2023-10-20T03:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.927074+0000", 2023-10-20T03:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.927074+0000", 2023-10-20T03:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.927075+0000", 2023-10-20T03:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2023-10-20T03:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.927312+0000", 2023-10-20T03:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023653300000000001 2023-10-20T03:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.927352+0000", 2023-10-20T03:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0241999999999999e-05 2023-10-20T03:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.927375+0000", 2023-10-20T03:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3027000000000002e-05 2023-10-20T03:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.929217+0000", 2023-10-20T03:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018416179999999999 2023-10-20T03:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.929249+0000", 2023-10-20T03:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1680000000000002e-05 2023-10-20T03:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.942950+0000", 2023-10-20T03:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013701186000000001 2023-10-20T03:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.943052+0000", 2023-10-20T03:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010268399999999999 2023-10-20T03:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.3s6 70/56 ECSubWrite(tid=55, reqid=client.4570.0:556, at_version=70'40, trim_to=0'0, roll_forward_to=60'38))", 2023-10-20T03:35:57.626 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.966857+0000", 2023-10-20T03:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.4567560669999997, 2023-10-20T03:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017176543999999998, 2023-10-20T03:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.627 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966857+0000", 2023-10-20T03:35:57.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966857+0000", 2023-10-20T03:35:57.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966858+0000", 2023-10-20T03:35:57.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2699999999999999e-07 2023-10-20T03:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967036+0000", 2023-10-20T03:35:57.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017807700000000001 2023-10-20T03:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967064+0000", 2023-10-20T03:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8748999999999999e-05 2023-10-20T03:35:57.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967087+0000", 2023-10-20T03:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2240000000000001e-05 2023-10-20T03:35:57.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967329+0000", 2023-10-20T03:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024282899999999999 2023-10-20T03:35:57.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967360+0000", 2023-10-20T03:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0094e-05 2023-10-20T03:35:57.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.983937+0000", 2023-10-20T03:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016577881999999999 2023-10-20T03:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.984034+0000", 2023-10-20T03:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6145999999999997e-05 2023-10-20T03:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.637 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.3s6 71/56 ECSubWrite(tid=63, reqid=client.4570.0:575, at_version=71'46, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:35:57.638 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.984211+0000", 2023-10-20T03:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.4394019140000003, 2023-10-20T03:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015632482999999999, 2023-10-20T03:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.639 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984211+0000", 2023-10-20T03:35:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984211+0000", 2023-10-20T03:35:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984213+0000", 2023-10-20T03:35:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.88e-06 2023-10-20T03:35:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984413+0000", 2023-10-20T03:35:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000199967 2023-10-20T03:35:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984451+0000", 2023-10-20T03:35:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8278999999999997e-05 2023-10-20T03:35:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984469+0000", 2023-10-20T03:35:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8033000000000002e-05 2023-10-20T03:35:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984652+0000", 2023-10-20T03:35:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018246700000000001 2023-10-20T03:35:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984685+0000", 2023-10-20T03:35:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2972999999999998e-05 2023-10-20T03:35:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.999764+0000", 2023-10-20T03:35:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015079298 2023-10-20T03:35:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.999844+0000", 2023-10-20T03:35:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9585999999999993e-05 2023-10-20T03:35:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es1 71/31 ECSubWrite(tid=52, reqid=client.4570.0:589, at_version=71'24, trim_to=0'0, roll_forward_to=61'23))", 2023-10-20T03:35:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:53.004322+0000", 2023-10-20T03:35:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.4192914869999997, 2023-10-20T03:35:57.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044306533000000002, 2023-10-20T03:35:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004322+0000", 2023-10-20T03:35:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004322+0000", 2023-10-20T03:35:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004324+0000", 2023-10-20T03:35:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.187e-06 2023-10-20T03:35:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004869+0000", 2023-10-20T03:35:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054558099999999995 2023-10-20T03:35:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004900+0000", 2023-10-20T03:35:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0366999999999999e-05 2023-10-20T03:35:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004937+0000", 2023-10-20T03:35:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7509000000000003e-05 2023-10-20T03:35:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.005151+0000", 2023-10-20T03:35:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021337 2023-10-20T03:35:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.005177+0000", 2023-10-20T03:35:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6650000000000001e-05 2023-10-20T03:35:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.048582+0000", 2023-10-20T03:35:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043404953000000003 2023-10-20T03:35:57.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.048628+0000", 2023-10-20T03:35:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5915999999999999e-05 2023-10-20T03:35:57.661 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.662 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.3s6 71/56 ECSubWrite(tid=64, reqid=client.4570.0:576, at_version=71'47, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:35:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:53.007051+0000", 2023-10-20T03:35:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.4165626260000002, 2023-10-20T03:35:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023764014999999999, 2023-10-20T03:35:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.663 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007051+0000", 2023-10-20T03:35:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007051+0000", 2023-10-20T03:35:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007053+0000", 2023-10-20T03:35:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.356e-06 2023-10-20T03:35:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007379+0000", 2023-10-20T03:35:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000326612 2023-10-20T03:35:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007419+0000", 2023-10-20T03:35:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9943e-05 2023-10-20T03:35:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007443+0000", 2023-10-20T03:35:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3436000000000001e-05 2023-10-20T03:35:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007617+0000", 2023-10-20T03:35:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000173804 2023-10-20T03:35:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007643+0000", 2023-10-20T03:35:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6720000000000002e-05 2023-10-20T03:35:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.030749+0000", 2023-10-20T03:35:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023105173999999999 2023-10-20T03:35:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.030815+0000", 2023-10-20T03:35:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5969999999999993e-05 2023-10-20T03:35:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.674 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es1 71/31 ECSubWrite(tid=53, reqid=client.4570.0:590, at_version=71'25, trim_to=0'0, roll_forward_to=61'23))", 2023-10-20T03:35:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:53.014260+0000", 2023-10-20T03:35:57.675 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.4093530129999996, 2023-10-20T03:35:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034398564999999999, 2023-10-20T03:35:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:35:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:35:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:35:57.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:35:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014260+0000", 2023-10-20T03:35:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:35:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014260+0000", 2023-10-20T03:35:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:35:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:35:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014262+0000", 2023-10-20T03:35:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8470000000000001e-06 2023-10-20T03:35:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:35:57.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014803+0000", 2023-10-20T03:35:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054058100000000005 2023-10-20T03:35:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:35:57.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014844+0000", 2023-10-20T03:35:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1383e-05 2023-10-20T03:35:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:35:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014867+0000", 2023-10-20T03:35:57.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3116e-05 2023-10-20T03:35:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:35:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.018959+0000", 2023-10-20T03:35:57.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040918090000000001 2023-10-20T03:35:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:35:57.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.018986+0000", 2023-10-20T03:35:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7353999999999999e-05 2023-10-20T03:35:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:35:57.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.048635+0000", 2023-10-20T03:35:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029648364 2023-10-20T03:35:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:35:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:35:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:35:57.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.048659+0000", 2023-10-20T03:35:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4111e-05 2023-10-20T03:35:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:35:57.686 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:35:57.687 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:35:57.688 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-10-20T03:35:57.689 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:57.684+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:57.752 INFO:tasks.rados.rados.0.smithi170.stdout:262: done (4 left) 2023-10-20T03:35:57.753 INFO:tasks.rados.rados.0.smithi170.stdout:263: snap_create 2023-10-20T03:35:57.753 INFO:tasks.rados.rados.0.smithi170.stdout:258: expect (ObjNum 43 snap 0 seq_num 43) 2023-10-20T03:35:57.788 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:57.785+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:57.797 INFO:tasks.rados.rados.0.smithi170.stdout:259: expect (ObjNum 2 snap 0 seq_num 2) 2023-10-20T03:35:57.802 INFO:tasks.rados.rados.0.smithi170.stdout:260: expect (ObjNum 111 snap 18 seq_num 111) 2023-10-20T03:35:57.824 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:57.824 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:57.824 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2023-10-20T03:35:57.824 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:57.838 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-10-20T03:35:57.890 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:57.884+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:57.978 INFO:tasks.rados.rados.0.smithi170.stdout:261: finishing copy_from racing read to smithi17039300-46 2023-10-20T03:35:57.978 INFO:tasks.rados.rados.0.smithi170.stdout:261: finishing copy_from to smithi17039300-46 2023-10-20T03:35:57.978 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 46 v 66 (ObjNum 105 snap 16 seq_num 105) dirty exists 2023-10-20T03:35:57.991 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:57.988+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:58.004 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:58.004 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:58.004 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:35:58.004 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:35:58.005 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:58.018 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-10-20T03:35:58.092 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:58.089+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:58.143 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:58.143 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2023-10-20T03:35:58.143 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2023-10-20T03:35:58.144 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2023-10-20T03:35:58.144 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.144 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:323 3.0s0 3:0e4cfe35:::smithi17039300-49:head [append 0~589824] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:58.144 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.357685+0000", 2023-10-20T03:35:58.144 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.782164351999999, 2023-10-20T03:35:58.145 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039449378, 2023-10-20T03:35:58.145 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.145 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:58.145 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.146 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.146 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.146 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 323 2023-10-20T03:35:58.146 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.147 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.147 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.147 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.147 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.357685+0000", 2023-10-20T03:35:58.147 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.148 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.148 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.148 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.148 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.357685+0000", 2023-10-20T03:35:58.149 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.149 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.149 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.149 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.149 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.357688+0000", 2023-10-20T03:35:58.150 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7159999999999999e-06 2023-10-20T03:35:58.150 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.150 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.150 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.151 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358334+0000", 2023-10-20T03:35:58.151 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00064636299999999995 2023-10-20T03:35:58.151 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.151 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.151 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.152 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358368+0000", 2023-10-20T03:35:58.152 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3182000000000002e-05 2023-10-20T03:35:58.152 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.152 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.152 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.153 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358382+0000", 2023-10-20T03:35:58.153 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4797000000000001e-05 2023-10-20T03:35:58.153 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.153 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.154 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.154 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358559+0000", 2023-10-20T03:35:58.154 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000177057 2023-10-20T03:35:58.154 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.155 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.155 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.155 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358736+0000", 2023-10-20T03:35:58.155 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000176861 2023-10-20T03:35:58.155 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.156 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.156 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.156 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361012+0000", 2023-10-20T03:35:58.156 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002275792 2023-10-20T03:35:58.157 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.157 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.157 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.157 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.396529+0000", 2023-10-20T03:35:58.157 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.035517254999999998 2023-10-20T03:35:58.158 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.158 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.158 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:58.158 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.396672+0000", 2023-10-20T03:35:58.159 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014286600000000001 2023-10-20T03:35:58.159 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.159 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.159 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.159 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.397135+0000", 2023-10-20T03:35:58.160 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00046248900000000001 2023-10-20T03:35:58.160 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.160 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.160 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.160 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.160 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.161 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:324 3.0s0 3:0e4cfe35:::smithi17039300-49:head [append 0~573440] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:58.161 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.365103+0000", 2023-10-20T03:35:58.161 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.774746568000001, 2023-10-20T03:35:58.161 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.045538596000000001, 2023-10-20T03:35:58.161 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.162 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:58.162 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.162 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.162 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.162 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 324 2023-10-20T03:35:58.163 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.163 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.163 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.163 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.163 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365103+0000", 2023-10-20T03:35:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365103+0000", 2023-10-20T03:35:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365105+0000", 2023-10-20T03:35:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3e-06 2023-10-20T03:35:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365744+0000", 2023-10-20T03:35:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000638213 2023-10-20T03:35:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365777+0000", 2023-10-20T03:35:58.169 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3522000000000002e-05 2023-10-20T03:35:58.169 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.169 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.169 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365792+0000", 2023-10-20T03:35:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5227e-05 2023-10-20T03:35:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.379267+0000", 2023-10-20T03:35:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013474832000000001 2023-10-20T03:35:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.379462+0000", 2023-10-20T03:35:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019448100000000001 2023-10-20T03:35:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.381551+0000", 2023-10-20T03:35:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002089034 2023-10-20T03:35:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.404299+0000", 2023-10-20T03:35:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.022748244000000001 2023-10-20T03:35:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.410190+0000", 2023-10-20T03:35:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0058907630000000002 2023-10-20T03:35:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.410642+0000", 2023-10-20T03:35:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00045197999999999999 2023-10-20T03:35:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:325 3.0s0 3:0e4cfe35:::smithi17039300-49:head [append 0~196608] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.367541+0000", 2023-10-20T03:35:58.179 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.772308415000001, 2023-10-20T03:35:58.179 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.044806372999999997, 2023-10-20T03:35:58.179 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.179 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:58.180 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.180 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.180 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.180 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 325 2023-10-20T03:35:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367541+0000", 2023-10-20T03:35:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.183 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367541+0000", 2023-10-20T03:35:58.183 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.183 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.183 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.184 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.184 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367544+0000", 2023-10-20T03:35:58.184 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6129999999999998e-06 2023-10-20T03:35:58.184 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.184 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.184 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.185 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367692+0000", 2023-10-20T03:35:58.185 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014814900000000001 2023-10-20T03:35:58.185 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.185 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367725+0000", 2023-10-20T03:35:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2752000000000001e-05 2023-10-20T03:35:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367759+0000", 2023-10-20T03:35:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4654999999999999e-05 2023-10-20T03:35:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.188 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.188 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.397445+0000", 2023-10-20T03:35:58.188 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029685547 2023-10-20T03:35:58.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.188 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.397646+0000", 2023-10-20T03:35:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020082300000000001 2023-10-20T03:35:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:35:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001263 2023-10-20T03:35:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.405032+0000", 2023-10-20T03:35:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0061230240000000003 2023-10-20T03:35:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:58.192 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.412065+0000", 2023-10-20T03:35:58.192 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.007033437 2023-10-20T03:35:58.192 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.192 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.412348+0000", 2023-10-20T03:35:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00028237299999999999 2023-10-20T03:35:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:326 3.0s0 3:0e4cfe35:::smithi17039300-49:head [setxattr _header (56) in=7b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.367791+0000", 2023-10-20T03:35:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.772059057, 2023-10-20T03:35:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.045708658999999999, 2023-10-20T03:35:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.196 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:58.196 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.196 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.196 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.196 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 326 2023-10-20T03:35:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367791+0000", 2023-10-20T03:35:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367791+0000", 2023-10-20T03:35:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367792+0000", 2023-10-20T03:35:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1370000000000001e-06 2023-10-20T03:35:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367798+0000", 2023-10-20T03:35:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1229999999999997e-06 2023-10-20T03:35:58.202 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.202 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.202 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.202 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367813+0000", 2023-10-20T03:35:58.203 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5130000000000001e-05 2023-10-20T03:35:58.203 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.203 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.203 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.203 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367823+0000", 2023-10-20T03:35:58.204 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0103e-05 2023-10-20T03:35:58.204 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.204 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.204 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402753+0000", 2023-10-20T03:35:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.034930226000000002 2023-10-20T03:35:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402936+0000", 2023-10-20T03:35:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000182542 2023-10-20T03:35:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.403933+0000", 2023-10-20T03:35:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00099678799999999997 2023-10-20T03:35:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.405527+0000", 2023-10-20T03:35:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001594332 2023-10-20T03:35:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.209 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:58.209 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.413173+0000", 2023-10-20T03:35:58.209 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0076458400000000001 2023-10-20T03:35:58.209 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.209 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.210 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.210 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.413499+0000", 2023-10-20T03:35:58.210 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00032643800000000002 2023-10-20T03:35:58.210 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.211 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.211 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.211 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.211 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.211 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:327 3.0s0 3:0e4cfe35:::smithi17039300-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:35:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.367835+0000", 2023-10-20T03:35:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.772014300999999, 2023-10-20T03:35:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.053408074999999999, 2023-10-20T03:35:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 327 2023-10-20T03:35:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367835+0000", 2023-10-20T03:35:58.215 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.215 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.215 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.215 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.215 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367835+0000", 2023-10-20T03:35:58.216 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.216 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.216 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.216 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367836+0000", 2023-10-20T03:35:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3699999999999999e-07 2023-10-20T03:35:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367840+0000", 2023-10-20T03:35:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6200000000000001e-06 2023-10-20T03:35:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367851+0000", 2023-10-20T03:35:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0764999999999999e-05 2023-10-20T03:35:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367858+0000", 2023-10-20T03:35:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7700000000000001e-06 2023-10-20T03:35:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.404384+0000", 2023-10-20T03:35:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.036525558999999999 2023-10-20T03:35:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.404538+0000", 2023-10-20T03:35:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000154 2023-10-20T03:35:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.414622+0000", 2023-10-20T03:35:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.01008392 2023-10-20T03:35:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.414717+0000", 2023-10-20T03:35:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5476999999999994e-05 2023-10-20T03:35:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.421243+0000", 2023-10-20T03:35:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0065260270000000002 2023-10-20T03:35:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:390 3.0s0 3:0e4cfe35:::smithi17039300-49:head [copy-from ver 15] snapc f=[f,d,b,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:35:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:31.382452+0000", 2023-10-20T03:35:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 26.757397974, 2023-10-20T03:35:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.059495292999999998, 2023-10-20T03:35:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 390 2023-10-20T03:35:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382452+0000", 2023-10-20T03:35:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382452+0000", 2023-10-20T03:35:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382455+0000", 2023-10-20T03:35:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4070000000000002e-06 2023-10-20T03:35:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382474+0000", 2023-10-20T03:35:58.233 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8422000000000001e-05 2023-10-20T03:35:58.233 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.233 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.233 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.234 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382516+0000", 2023-10-20T03:35:58.234 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2376000000000002e-05 2023-10-20T03:35:58.234 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.234 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.234 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.235 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382534+0000", 2023-10-20T03:35:58.235 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8335999999999999e-05 2023-10-20T03:35:58.235 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.235 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382662+0000", 2023-10-20T03:35:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000128083 2023-10-20T03:35:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op must wait for map", 2023-10-20T03:35:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382676+0000", 2023-10-20T03:35:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3679999999999999e-05 2023-10-20T03:35:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.387857+0000", 2023-10-20T03:35:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0051807490000000001 2023-10-20T03:35:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.388142+0000", 2023-10-20T03:35:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000284729 2023-10-20T03:35:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.413968+0000", 2023-10-20T03:35:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025826947999999999 2023-10-20T03:35:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.440223+0000", 2023-10-20T03:35:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026254116000000001 2023-10-20T03:35:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.441418+0000", 2023-10-20T03:35:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001195113 2023-10-20T03:35:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.441947+0000", 2023-10-20T03:35:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00052933400000000001 2023-10-20T03:35:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:391 3.0s0 3:0e4cfe35:::smithi17039300-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:35:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:31.382572+0000", 2023-10-20T03:35:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 26.757277740999999, 2023-10-20T03:35:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.056308376, 2023-10-20T03:35:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 391 2023-10-20T03:35:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382572+0000", 2023-10-20T03:35:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382572+0000", 2023-10-20T03:35:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382573+0000", 2023-10-20T03:35:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4130000000000001e-06 2023-10-20T03:35:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382580+0000", 2023-10-20T03:35:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0670000000000003e-06 2023-10-20T03:35:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382603+0000", 2023-10-20T03:35:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2099999999999998e-05 2023-10-20T03:35:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382612+0000", 2023-10-20T03:35:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3519999999999996e-06 2023-10-20T03:35:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382806+0000", 2023-10-20T03:35:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019382199999999999 2023-10-20T03:35:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.258 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.258 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:58.258 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382827+0000", 2023-10-20T03:35:58.258 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1552e-05 2023-10-20T03:35:58.258 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.259 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.259 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.259 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.388779+0000", 2023-10-20T03:35:58.259 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0059519960000000002 2023-10-20T03:35:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for blocked object", 2023-10-20T03:35:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.388904+0000", 2023-10-20T03:35:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000124893 2023-10-20T03:35:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.438358+0000", 2023-10-20T03:35:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.049453814999999998 2023-10-20T03:35:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.438533+0000", 2023-10-20T03:35:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017465199999999999 2023-10-20T03:35:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.438880+0000", 2023-10-20T03:35:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000347714 2023-10-20T03:35:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:408 3.fs1 3:f90c0d61:::smithi17039300-26:head [append 0~753664] snapc f=[d,3,2] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:35:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.443873+0000", 2023-10-20T03:35:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.695976883, 2023-10-20T03:35:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.045012336, 2023-10-20T03:35:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 408 2023-10-20T03:35:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.443873+0000", 2023-10-20T03:35:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.443873+0000", 2023-10-20T03:35:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.443876+0000", 2023-10-20T03:35:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8200000000000001e-06 2023-10-20T03:35:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.444508+0000", 2023-10-20T03:35:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00063252700000000002 2023-10-20T03:35:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.444542+0000", 2023-10-20T03:35:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4292999999999998e-05 2023-10-20T03:35:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.444564+0000", 2023-10-20T03:35:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1066000000000001e-05 2023-10-20T03:35:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.446120+0000", 2023-10-20T03:35:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0015568089999999999 2023-10-20T03:35:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.446540+0000", 2023-10-20T03:35:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00041965900000000002 2023-10-20T03:35:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.449079+0000", 2023-10-20T03:35:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0025392610000000001 2023-10-20T03:35:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.469859+0000", 2023-10-20T03:35:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.020779766000000002 2023-10-20T03:35:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.488709+0000", 2023-10-20T03:35:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.018849798000000001 2023-10-20T03:35:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.488885+0000", 2023-10-20T03:35:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017633699999999999 2023-10-20T03:35:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:409 3.fs1 3:f90c0d61:::smithi17039300-26:head [append 0~720896] snapc f=[d,3,2] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:35:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.456955+0000", 2023-10-20T03:35:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.682894825999998, 2023-10-20T03:35:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039289445999999999, 2023-10-20T03:35:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 409 2023-10-20T03:35:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.456955+0000", 2023-10-20T03:35:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.456955+0000", 2023-10-20T03:35:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.456958+0000", 2023-10-20T03:35:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.253e-06 2023-10-20T03:35:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.457987+0000", 2023-10-20T03:35:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0010284510000000001 2023-10-20T03:35:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458025+0000", 2023-10-20T03:35:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8566000000000003e-05 2023-10-20T03:35:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458043+0000", 2023-10-20T03:35:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.783e-05 2023-10-20T03:35:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.470966+0000", 2023-10-20T03:35:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012923244 2023-10-20T03:35:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.471149+0000", 2023-10-20T03:35:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000182638 2023-10-20T03:35:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.472881+0000", 2023-10-20T03:35:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017317610000000001 2023-10-20T03:35:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.296 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.296 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.296 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.488407+0000", 2023-10-20T03:35:58.296 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015526521 2023-10-20T03:35:58.296 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.297 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:58.297 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.496153+0000", 2023-10-20T03:35:58.297 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0077454489999999997 2023-10-20T03:35:58.298 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.298 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.298 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.298 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.496244+0000", 2023-10-20T03:35:58.298 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1732999999999997e-05 2023-10-20T03:35:58.299 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.299 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.299 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.299 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.299 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.300 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:410 3.fs1 3:f90c0d61:::smithi17039300-26:head [append 0~49152] snapc f=[d,3,2] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:35:58.300 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.458073+0000", 2023-10-20T03:35:58.300 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.681776640999999, 2023-10-20T03:35:58.300 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039280043000000001, 2023-10-20T03:35:58.301 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.301 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:58.301 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.301 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.302 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.302 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 410 2023-10-20T03:35:58.302 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.302 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.302 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.303 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.303 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458073+0000", 2023-10-20T03:35:58.303 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.303 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.304 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.304 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458073+0000", 2023-10-20T03:35:58.304 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.305 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.305 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.305 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.305 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458074+0000", 2023-10-20T03:35:58.306 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2100000000000001e-06 2023-10-20T03:35:58.306 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.306 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.306 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.306 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458151+0000", 2023-10-20T03:35:58.307 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6542e-05 2023-10-20T03:35:58.307 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.307 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.307 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.307 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458168+0000", 2023-10-20T03:35:58.308 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7552999999999999e-05 2023-10-20T03:35:58.308 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.308 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.308 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.309 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458180+0000", 2023-10-20T03:35:58.309 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1826e-05 2023-10-20T03:35:58.309 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.309 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.309 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.310 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.485589+0000", 2023-10-20T03:35:58.310 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027409103000000001 2023-10-20T03:35:58.310 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.310 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.310 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.311 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.485726+0000", 2023-10-20T03:35:58.311 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000136482 2023-10-20T03:35:58.311 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.311 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.311 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.312 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.486676+0000", 2023-10-20T03:35:58.312 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000949766 2023-10-20T03:35:58.312 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.312 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.312 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.313 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.491210+0000", 2023-10-20T03:35:58.313 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0045347230000000001 2023-10-20T03:35:58.313 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.313 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.313 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:58.313 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.497284+0000", 2023-10-20T03:35:58.314 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0060733059999999997 2023-10-20T03:35:58.314 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.314 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.315 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.315 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.497353+0000", 2023-10-20T03:35:58.315 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9531999999999998e-05 2023-10-20T03:35:58.315 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.315 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.316 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.316 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.316 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.316 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:411 3.fs1 3:f90c0d61:::smithi17039300-26:head [setxattr _header (57) in=7b] snapc f=[d,3,2] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:35:58.316 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.458196+0000", 2023-10-20T03:35:58.317 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.681653740000002, 2023-10-20T03:35:58.317 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039654613999999998, 2023-10-20T03:35:58.317 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.317 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:58.317 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.317 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.318 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.318 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 411 2023-10-20T03:35:58.318 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.318 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.318 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.319 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.319 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458196+0000", 2023-10-20T03:35:58.319 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.319 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.320 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.320 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.320 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458196+0000", 2023-10-20T03:35:58.320 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.321 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.321 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.321 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.321 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458197+0000", 2023-10-20T03:35:58.321 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1299999999999998e-07 2023-10-20T03:35:58.322 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.322 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.322 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.322 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458202+0000", 2023-10-20T03:35:58.323 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2170000000000002e-06 2023-10-20T03:35:58.323 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.323 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.323 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.324 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458212+0000", 2023-10-20T03:35:58.324 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0000000000000001e-05 2023-10-20T03:35:58.324 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.324 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.325 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.325 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458220+0000", 2023-10-20T03:35:58.325 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4560000000000003e-06 2023-10-20T03:35:58.325 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.326 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.326 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.326 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.489002+0000", 2023-10-20T03:35:58.326 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030782868000000001 2023-10-20T03:35:58.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.327 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.327 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.327 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.489245+0000", 2023-10-20T03:35:58.327 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024254100000000001 2023-10-20T03:35:58.327 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.328 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.328 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.328 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.490266+0000", 2023-10-20T03:35:58.328 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001020721 2023-10-20T03:35:58.328 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.329 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.329 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.329 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.491971+0000", 2023-10-20T03:35:58.329 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017054940000000001 2023-10-20T03:35:58.329 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.330 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.330 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:58.330 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.497768+0000", 2023-10-20T03:35:58.330 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0057964269999999998 2023-10-20T03:35:58.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.331 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.331 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.331 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.497851+0000", 2023-10-20T03:35:58.331 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2977000000000002e-05 2023-10-20T03:35:58.332 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.332 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.332 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.333 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.333 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.333 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:412 3.fs1 3:f90c0d61:::smithi17039300-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:35:58.333 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.458230+0000", 2023-10-20T03:35:58.334 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.681619874999999, 2023-10-20T03:35:58.334 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.042333239000000002, 2023-10-20T03:35:58.334 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.334 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:58.334 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.335 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.335 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.335 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 412 2023-10-20T03:35:58.335 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.336 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.336 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.336 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.336 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458230+0000", 2023-10-20T03:35:58.336 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.337 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.337 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.337 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.337 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458230+0000", 2023-10-20T03:35:58.337 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.338 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.338 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.338 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.338 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458231+0000", 2023-10-20T03:35:58.338 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8700000000000005e-07 2023-10-20T03:35:58.339 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.339 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.339 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.339 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458234+0000", 2023-10-20T03:35:58.339 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5499999999999999e-06 2023-10-20T03:35:58.340 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.340 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.340 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.340 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458245+0000", 2023-10-20T03:35:58.341 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0716999999999999e-05 2023-10-20T03:35:58.341 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.341 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.341 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.341 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458252+0000", 2023-10-20T03:35:58.342 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.486e-06 2023-10-20T03:35:58.342 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.342 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.342 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.342 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.491656+0000", 2023-10-20T03:35:58.343 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.033403571 2023-10-20T03:35:58.343 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.343 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.343 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:58.343 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.491757+0000", 2023-10-20T03:35:58.344 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101383 2023-10-20T03:35:58.344 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.344 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.344 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.344 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.498257+0000", 2023-10-20T03:35:58.345 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0064998809999999999 2023-10-20T03:35:58.345 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.345 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.345 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.345 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.498357+0000", 2023-10-20T03:35:58.346 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9491e-05 2023-10-20T03:35:58.346 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.346 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.346 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.346 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.500563+0000", 2023-10-20T03:35:58.347 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0022063730000000002 2023-10-20T03:35:58.347 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.347 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.347 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.347 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.348 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.348 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:421 3.0s0 3:0e4cfe35:::smithi17039300-49:head [setxattr _header (57) in=7b] snapc f=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-10-20T03:35:58.348 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.469730+0000", 2023-10-20T03:35:58.348 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.670120097000002, 2023-10-20T03:35:58.348 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.033931731999999999, 2023-10-20T03:35:58.349 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.349 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:58.349 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.349 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.350 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.350 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 421 2023-10-20T03:35:58.350 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.350 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.351 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.351 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.351 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469730+0000", 2023-10-20T03:35:58.351 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.351 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.352 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.352 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.352 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469730+0000", 2023-10-20T03:35:58.352 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.352 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.353 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.353 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.353 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469731+0000", 2023-10-20T03:35:58.354 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5799999999999999e-06 2023-10-20T03:35:58.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.354 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.355 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.355 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469741+0000", 2023-10-20T03:35:58.355 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6700000000000006e-06 2023-10-20T03:35:58.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.356 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.356 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.356 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469765+0000", 2023-10-20T03:35:58.357 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4216e-05 2023-10-20T03:35:58.357 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.358 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.358 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.359 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469780+0000", 2023-10-20T03:35:58.359 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.482e-05 2023-10-20T03:35:58.359 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.360 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.360 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.360 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.496944+0000", 2023-10-20T03:35:58.360 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027164562999999999 2023-10-20T03:35:58.361 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.361 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.362 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.362 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.497153+0000", 2023-10-20T03:35:58.362 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020828900000000001 2023-10-20T03:35:58.362 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.362 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.363 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.363 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.498227+0000", 2023-10-20T03:35:58.363 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0010742670000000001 2023-10-20T03:35:58.363 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.364 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.364 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.364 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.501137+0000", 2023-10-20T03:35:58.364 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0029095969999999999 2023-10-20T03:35:58.365 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.365 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.365 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:58.365 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.503446+0000", 2023-10-20T03:35:58.366 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0023098580000000001 2023-10-20T03:35:58.366 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.366 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.367 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.367 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.503661+0000", 2023-10-20T03:35:58.367 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00021487200000000001 2023-10-20T03:35:58.368 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.368 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.368 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.368 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.369 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.369 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:422 3.0s0 3:0e4cfe35:::smithi17039300-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2023-10-20T03:35:58.369 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.469801+0000", 2023-10-20T03:35:58.369 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.670049059, 2023-10-20T03:35:58.369 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.036242191, 2023-10-20T03:35:58.370 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.370 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:58.370 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.370 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.370 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.371 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 422 2023-10-20T03:35:58.371 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.371 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.371 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.371 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.372 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469801+0000", 2023-10-20T03:35:58.372 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.372 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.372 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.372 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.373 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469801+0000", 2023-10-20T03:35:58.373 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.373 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.373 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.374 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.374 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469802+0000", 2023-10-20T03:35:58.374 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.19e-06 2023-10-20T03:35:58.374 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.375 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.375 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.375 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469809+0000", 2023-10-20T03:35:58.375 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4630000000000004e-06 2023-10-20T03:35:58.376 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.376 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.376 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.376 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469832+0000", 2023-10-20T03:35:58.376 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2883000000000001e-05 2023-10-20T03:35:58.377 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.377 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.377 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.377 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469846+0000", 2023-10-20T03:35:58.377 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4113e-05 2023-10-20T03:35:58.378 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.378 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.378 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.378 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.500041+0000", 2023-10-20T03:35:58.379 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030195161000000002 2023-10-20T03:35:58.379 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.379 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.379 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:35:58.379 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.500396+0000", 2023-10-20T03:35:58.380 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00035421399999999999 2023-10-20T03:35:58.380 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.380 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.380 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.381 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.504208+0000", 2023-10-20T03:35:58.381 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0038123390000000001 2023-10-20T03:35:58.381 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.381 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.382 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.382 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.504283+0000", 2023-10-20T03:35:58.382 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4981999999999995e-05 2023-10-20T03:35:58.382 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.382 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.383 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.383 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.506043+0000", 2023-10-20T03:35:58.383 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001759846 2023-10-20T03:35:58.383 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.384 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.384 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.384 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.385 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.385 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.1s5 60/31 ECSubWrite(tid=190, reqid=client.4570.0:452, at_version=60'54, trim_to=0'0, roll_forward_to=58'48))", 2023-10-20T03:35:58.385 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.861666+0000", 2023-10-20T03:35:58.385 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 16.278184106000001, 2023-10-20T03:35:58.385 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.037670670000000003, 2023-10-20T03:35:58.386 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.386 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:58.386 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.387 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.387 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.387 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.861666+0000", 2023-10-20T03:35:58.387 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.387 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.388 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.388 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.388 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.861666+0000", 2023-10-20T03:35:58.388 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.388 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.388 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.389 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.389 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.861692+0000", 2023-10-20T03:35:58.389 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6239000000000001e-05 2023-10-20T03:35:58.389 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.390 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.390 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.390 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.862699+0000", 2023-10-20T03:35:58.390 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0010070540000000001 2023-10-20T03:35:58.390 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.390 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.391 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.391 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.862728+0000", 2023-10-20T03:35:58.391 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8626e-05 2023-10-20T03:35:58.391 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.391 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.392 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.392 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.862748+0000", 2023-10-20T03:35:58.392 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0199000000000001e-05 2023-10-20T03:35:58.392 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.393 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.393 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.393 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.862946+0000", 2023-10-20T03:35:58.393 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019865900000000001 2023-10-20T03:35:58.393 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.394 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.394 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.394 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.862974+0000", 2023-10-20T03:35:58.394 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7616000000000001e-05 2023-10-20T03:35:58.394 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.395 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.395 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.395 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.899243+0000", 2023-10-20T03:35:58.395 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.03626923 2023-10-20T03:35:58.395 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.396 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.396 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.396 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.899336+0000", 2023-10-20T03:35:58.396 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3047000000000005e-05 2023-10-20T03:35:58.396 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.397 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.397 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.397 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.397 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.397 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:552 3.0s0 3:083aab2b:::smithi17039300-2:head [copy-from ver 11] snapc 23=[23,21] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2023-10-20T03:35:58.398 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:50.925989+0000", 2023-10-20T03:35:58.398 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 7.2138604639999997, 2023-10-20T03:35:58.398 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.061771054999999998, 2023-10-20T03:35:58.398 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.398 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:58.399 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.399 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.399 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.399 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 552 2023-10-20T03:35:58.400 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.400 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.400 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.400 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.401 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.925989+0000", 2023-10-20T03:35:58.401 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.401 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.401 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.402 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.402 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.925989+0000", 2023-10-20T03:35:58.402 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.402 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.403 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.403 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.403 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.925992+0000", 2023-10-20T03:35:58.403 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0699999999999998e-06 2023-10-20T03:35:58.403 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.404 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.404 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.404 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926010+0000", 2023-10-20T03:35:58.405 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8099999999999999e-05 2023-10-20T03:35:58.405 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.405 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.405 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.406 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926045+0000", 2023-10-20T03:35:58.406 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5089000000000002e-05 2023-10-20T03:35:58.406 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.406 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.407 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.407 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926062+0000", 2023-10-20T03:35:58.407 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6897000000000001e-05 2023-10-20T03:35:58.407 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.408 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.408 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.408 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926214+0000", 2023-10-20T03:35:58.408 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015152699999999999 2023-10-20T03:35:58.409 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.409 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.409 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op must wait for map", 2023-10-20T03:35:58.409 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926233+0000", 2023-10-20T03:35:58.410 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8949000000000001e-05 2023-10-20T03:35:58.410 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.410 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.410 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.411 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.931605+0000", 2023-10-20T03:35:58.411 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0053720529999999999 2023-10-20T03:35:58.411 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.411 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.411 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.412 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.931794+0000", 2023-10-20T03:35:58.412 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018947300000000001 2023-10-20T03:35:58.412 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.412 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.413 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.413 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.953907+0000", 2023-10-20T03:35:58.413 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.022112526 2023-10-20T03:35:58.413 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.414 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.414 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.414 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.984803+0000", 2023-10-20T03:35:58.414 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030896078 2023-10-20T03:35:58.415 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.415 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.415 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:58.415 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.987358+0000", 2023-10-20T03:35:58.415 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0025545469999999999 2023-10-20T03:35:58.416 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.416 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.416 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.416 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.987760+0000", 2023-10-20T03:35:58.417 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000402746 2023-10-20T03:35:58.417 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.417 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.417 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.418 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.418 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.418 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:553 3.0s0 3:083aab2b:::smithi17039300-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2023-10-20T03:35:58.418 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:50.926100+0000", 2023-10-20T03:35:58.418 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 7.213749236, 2023-10-20T03:35:58.419 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.058318110999999999, 2023-10-20T03:35:58.419 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.419 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:58.419 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.420 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.420 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.420 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 553 2023-10-20T03:35:58.420 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.420 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.420 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.421 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.421 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926100+0000", 2023-10-20T03:35:58.421 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.422 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.422 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.422 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.422 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926100+0000", 2023-10-20T03:35:58.423 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.423 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.423 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.423 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.423 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926102+0000", 2023-10-20T03:35:58.424 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3769999999999999e-06 2023-10-20T03:35:58.424 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.424 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.424 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.424 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926109+0000", 2023-10-20T03:35:58.425 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8159999999999996e-06 2023-10-20T03:35:58.425 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.425 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.425 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.425 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926131+0000", 2023-10-20T03:35:58.426 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2370000000000001e-05 2023-10-20T03:35:58.426 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.426 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.426 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.427 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926147+0000", 2023-10-20T03:35:58.427 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6025999999999999e-05 2023-10-20T03:35:58.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.427 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.427 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.428 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926345+0000", 2023-10-20T03:35:58.428 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019841300000000001 2023-10-20T03:35:58.428 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.428 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.429 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:35:58.429 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926358+0000", 2023-10-20T03:35:58.429 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2262999999999999e-05 2023-10-20T03:35:58.429 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.430 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.430 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.430 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.932162+0000", 2023-10-20T03:35:58.430 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0058040820000000003 2023-10-20T03:35:58.431 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.431 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.431 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for blocked object", 2023-10-20T03:35:58.431 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.932353+0000", 2023-10-20T03:35:58.431 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019135300000000001 2023-10-20T03:35:58.432 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.432 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.432 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.432 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.983370+0000", 2023-10-20T03:35:58.432 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.051016367999999999 2023-10-20T03:35:58.433 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.433 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.433 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.433 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.983685+0000", 2023-10-20T03:35:58.434 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00031522700000000001 2023-10-20T03:35:58.434 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.434 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.434 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.434 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.984419+0000", 2023-10-20T03:35:58.435 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00073381599999999998 2023-10-20T03:35:58.435 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.435 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.435 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.435 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.436 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.436 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.es5 71/31 ECSubWrite(tid=52, reqid=client.4570.0:589, at_version=71'24, trim_to=0'0, roll_forward_to=61'23))", 2023-10-20T03:35:58.436 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.004276+0000", 2023-10-20T03:35:58.436 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 5.135573731, 2023-10-20T03:35:58.437 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.033989473999999999, 2023-10-20T03:35:58.437 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.437 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:35:58.437 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.437 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.438 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.438 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004276+0000", 2023-10-20T03:35:58.438 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.438 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.438 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.439 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.439 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004276+0000", 2023-10-20T03:35:58.439 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.440 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.440 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.440 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.440 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004276+0000", 2023-10-20T03:35:58.440 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4999999999999998e-07 2023-10-20T03:35:58.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.441 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.441 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.441 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004557+0000", 2023-10-20T03:35:58.441 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00028076400000000002 2023-10-20T03:35:58.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.442 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.442 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.442 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004588+0000", 2023-10-20T03:35:58.442 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0417e-05 2023-10-20T03:35:58.442 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004621+0000", 2023-10-20T03:35:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3416e-05 2023-10-20T03:35:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.443 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.005126+0000", 2023-10-20T03:35:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00050556099999999997 2023-10-20T03:35:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.444 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.005192+0000", 2023-10-20T03:35:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5429e-05 2023-10-20T03:35:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.445 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.038177+0000", 2023-10-20T03:35:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.032984861999999997 2023-10-20T03:35:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.446 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.447 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.038265+0000", 2023-10-20T03:35:58.447 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.8675000000000001e-05 2023-10-20T03:35:58.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.447 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.448 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.448 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.448 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:628 3.0s0 3:0ac18d01:::smithi17039300-46:head [copy-from ver 66] snapc 28=[28,26] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2023-10-20T03:35:58.448 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:57.010408+0000", 2023-10-20T03:35:58.448 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 1.129441554, 2023-10-20T03:35:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.069670394999999996, 2023-10-20T03:35:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:35:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.449 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 628 2023-10-20T03:35:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.450 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.451 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.451 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010408+0000", 2023-10-20T03:35:58.451 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.451 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.452 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.452 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.452 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010408+0000", 2023-10-20T03:35:58.452 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.452 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.453 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.453 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.453 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010411+0000", 2023-10-20T03:35:58.453 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.666e-06 2023-10-20T03:35:58.453 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.454 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.454 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.454 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010423+0000", 2023-10-20T03:35:58.454 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2466999999999999e-05 2023-10-20T03:35:58.455 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.455 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.455 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.455 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010458+0000", 2023-10-20T03:35:58.455 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4224000000000003e-05 2023-10-20T03:35:58.455 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.456 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.456 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.456 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010474+0000", 2023-10-20T03:35:58.456 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6325999999999999e-05 2023-10-20T03:35:58.457 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.457 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.457 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.457 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010589+0000", 2023-10-20T03:35:58.458 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011505700000000001 2023-10-20T03:35:58.458 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.458 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.458 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.459 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010780+0000", 2023-10-20T03:35:58.459 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000191246 2023-10-20T03:35:58.459 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.459 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.459 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:35:58.460 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.036134+0000", 2023-10-20T03:35:58.460 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025353844 2023-10-20T03:35:58.460 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.460 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.460 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:35:58.461 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.076512+0000", 2023-10-20T03:35:58.461 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.040378485999999998 2023-10-20T03:35:58.461 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.461 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.462 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:35:58.462 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.079728+0000", 2023-10-20T03:35:58.462 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0032154549999999999 2023-10-20T03:35:58.463 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.463 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.463 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.463 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.080079+0000", 2023-10-20T03:35:58.464 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00035062400000000001 2023-10-20T03:35:58.464 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.464 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.464 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.464 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.465 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.465 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:629 3.0s0 3:0ac18d01:::smithi17039300-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2023-10-20T03:35:58.465 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:57.010513+0000", 2023-10-20T03:35:58.465 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 1.129337193, 2023-10-20T03:35:58.465 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.065291919000000004, 2023-10-20T03:35:58.466 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:35:58.466 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:35:58.467 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:35:58.467 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:35:58.467 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:35:58.467 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 629 2023-10-20T03:35:58.468 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.468 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:35:58.468 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.469 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:35:58.469 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010513+0000", 2023-10-20T03:35:58.469 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.470 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.470 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.471 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:35:58.471 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010513+0000", 2023-10-20T03:35:58.472 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:35:58.472 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.472 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.473 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:35:58.473 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010514+0000", 2023-10-20T03:35:58.473 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0899999999999999e-06 2023-10-20T03:35:58.474 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.474 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.474 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:35:58.475 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010519+0000", 2023-10-20T03:35:58.475 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2630000000000003e-06 2023-10-20T03:35:58.475 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.475 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.476 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:35:58.476 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010532+0000", 2023-10-20T03:35:58.476 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2989e-05 2023-10-20T03:35:58.476 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.476 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.477 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:35:58.477 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010541+0000", 2023-10-20T03:35:58.477 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9129999999999992e-06 2023-10-20T03:35:58.477 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.477 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.478 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.478 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.011068+0000", 2023-10-20T03:35:58.478 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00052740100000000004 2023-10-20T03:35:58.478 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.478 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.479 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for blocked object", 2023-10-20T03:35:58.479 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.011176+0000", 2023-10-20T03:35:58.479 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010788700000000001 2023-10-20T03:35:58.479 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.480 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.480 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:35:58.480 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.075253+0000", 2023-10-20T03:35:58.480 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.064077032000000006 2023-10-20T03:35:58.480 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.481 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.481 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:35:58.481 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.075450+0000", 2023-10-20T03:35:58.481 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019677799999999999 2023-10-20T03:35:58.482 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:35:58.482 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:35:58.482 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:35:58.482 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.075804+0000", 2023-10-20T03:35:58.482 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00035456600000000001 2023-10-20T03:35:58.483 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.483 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.483 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.483 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:35:58.484 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:35:58.484 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:58.484 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:35:58.487 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-10-20T03:35:58.487 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:35:58.485+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:35:58.587 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:58.584+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:58.618 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:58.618 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:58.618 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2023-10-20T03:35:58.619 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:58.632 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-10-20T03:35:58.646 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:35:58.646 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.665 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:35:58.666 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.688 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:58.684+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:58.688 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:35:58.689 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.706 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:35:58.706 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.724 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:35:58.724 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.745 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:35:58.746 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.746 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:35:58.752 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:35:58.753 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:35:58.753 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:35:58.753 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:35:58.753 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:35:58.765 INFO:tasks.rados.rados.0.smithi170.stdout:258: done (4 left) 2023-10-20T03:35:58.765 INFO:tasks.rados.rados.0.smithi170.stdout:259: done (3 left) 2023-10-20T03:35:58.765 INFO:tasks.rados.rados.0.smithi170.stdout:260: done (2 left) 2023-10-20T03:35:58.765 INFO:tasks.rados.rados.0.smithi170.stdout:261: done (1 left) 2023-10-20T03:35:58.766 INFO:tasks.rados.rados.0.smithi170.stdout:263: done (0 left) 2023-10-20T03:35:58.766 INFO:tasks.rados.rados.0.smithi170.stdout:264: copy_from oid 23 from oid 1 current snap is 22 2023-10-20T03:35:58.766 INFO:tasks.rados.rados.0.smithi170.stdout:265: snap_remove snap 21 2023-10-20T03:35:58.767 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:35:58.768 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-10-20T03:35:58.788 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:35:58.789 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.789 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:58.788+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:58.809 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:35:58.810 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.830 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:35:58.831 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.850 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:35:58.851 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.868 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:35:58.869 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.889 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:58.885+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:58.890 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:35:58.891 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.893 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:58.893 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:58.908 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-10-20T03:35:58.912 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:35:58.912 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.932 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:35:58.933 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:35:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:35:58.939 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:35:58.990 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:35:58.988+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:35:59.034 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-10-20T03:35:59.073 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:59.073 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:59.087 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-10-20T03:35:59.131 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:35:59.129+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:35:59.191 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:35:59.189+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:35:59.204 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:59.204 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:59.218 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-10-20T03:35:59.292 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:35:59.288+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:35:59.380 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:59.380 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:59.393 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:35:59.388+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:35:59.394 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-10-20T03:35:59.493 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:35:59.488+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:35:59.515 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:59.515 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:59.530 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-10-20T03:35:59.594 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:35:59.593+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:35:59.652 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:59.652 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:59.666 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-10-20T03:35:59.694 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:35:59.692+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:35:59.761 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2023-10-20T03:35:59.769 INFO:tasks.rados.rados.0.smithi170.stdout:265: done (1 left) 2023-10-20T03:35:59.769 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 41 current snap is 22 2023-10-20T03:35:59.769 INFO:tasks.rados.rados.0.smithi170.stdout:264: finishing copy_from racing read to smithi17039300-23 2023-10-20T03:35:59.770 INFO:tasks.rados.rados.0.smithi170.stdout:264: finishing copy_from to smithi17039300-23 2023-10-20T03:35:59.770 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 23 v 34 (ObjNum 77 snap 5 seq_num 77) dirty exists 2023-10-20T03:35:59.770 INFO:tasks.rados.rados.0.smithi170.stdout:266: seq_num 116 ranges {0=786432,786432=458752,1245184=442368,1687552=262144} 2023-10-20T03:35:59.782 INFO:tasks.rados.rados.0.smithi170.stdout:266: writing smithi17039300-41 from 0 to 786432 tid 1 2023-10-20T03:35:59.790 INFO:tasks.rados.rados.0.smithi170.stdout:266: writing smithi17039300-41 from 786432 to 1245184 tid 2 2023-10-20T03:35:59.797 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:59.797 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:59.797 INFO:tasks.rados.rados.0.smithi170.stdout:266: writing smithi17039300-41 from 1245184 to 1687552 tid 3 2023-10-20T03:35:59.802 INFO:tasks.rados.rados.0.smithi170.stdout:266: writing smithi17039300-41 from 1687552 to 1949696 tid 4 2023-10-20T03:35:59.802 INFO:tasks.rados.rados.0.smithi170.stdout:264: done (1 left) 2023-10-20T03:35:59.802 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 36 current snap is 22 2023-10-20T03:35:59.802 INFO:tasks.rados.rados.0.smithi170.stdout:267: seq_num 117 ranges {1818624=425984,2244608=606208,2850816=540672,3391488=409600} 2023-10-20T03:35:59.812 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-10-20T03:35:59.831 INFO:tasks.rados.rados.0.smithi170.stdout:267: writing smithi17039300-36 from 1818624 to 2244608 tid 1 2023-10-20T03:35:59.834 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:35:59.833+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:35:59.839 INFO:tasks.rados.rados.0.smithi170.stdout:267: writing smithi17039300-36 from 2244608 to 2850816 tid 2 2023-10-20T03:35:59.845 INFO:tasks.rados.rados.0.smithi170.stdout:267: writing smithi17039300-36 from 2850816 to 3391488 tid 3 2023-10-20T03:35:59.849 INFO:tasks.rados.rados.0.smithi170.stdout:267: writing smithi17039300-36 from 3391488 to 3801088 tid 4 2023-10-20T03:35:59.849 INFO:tasks.rados.rados.0.smithi170.stdout:268: snap_remove snap 18 2023-10-20T03:35:59.895 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:35:59.893+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:35:59.965 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:35:59.966 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:35:59.980 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-10-20T03:36:00.001 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15] out_osds: [12] dead_osds: [6] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 10, 14, 3, 7, 11, 15] 2023-10-20T03:36:00.001 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-10-20T03:36:00.002 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2023-10-20T03:36:00.002 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-10-20T03:36:00.002 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-10-20T03:36:00.003 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2023-10-20T03:36:00.026 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:00.025+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:00.097 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:00.092+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:00.103 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:00.103 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:00.118 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-10-20T03:36:00.143 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:00.144 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-10-20T03:36:00.144 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:00.198 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:36:00.192+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:36:00.243 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:00.244 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:00.259 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-10-20T03:36:00.299 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:36:00.296+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:36:00.384 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:00.385 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:00.399 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-10-20T03:36:00.400 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:00.397+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:00.501 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:00.501+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:00.526 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:00.526 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:00.540 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-10-20T03:36:00.602 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:00.601+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:00.659 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:00.660 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:00.674 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-10-20T03:36:00.702 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:36:00.700+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:36:00.773 INFO:tasks.rados.rados.0.smithi170.stdout:268: done (2 left) 2023-10-20T03:36:00.773 INFO:tasks.rados.rados.0.smithi170.stdout:269: snap_create 2023-10-20T03:36:00.773 INFO:tasks.rados.rados.0.smithi170.stdout:266: finishing write tid 1 to smithi17039300-41 2023-10-20T03:36:00.773 INFO:tasks.rados.rados.0.smithi170.stdout:266: finishing write tid 2 to smithi17039300-41 2023-10-20T03:36:00.774 INFO:tasks.rados.rados.0.smithi170.stdout:266: finishing write tid 3 to smithi17039300-41 2023-10-20T03:36:00.774 INFO:tasks.rados.rados.0.smithi170.stdout:266: finishing write tid 4 to smithi17039300-41 2023-10-20T03:36:00.774 INFO:tasks.rados.rados.0.smithi170.stdout:266: finishing write tid 5 to smithi17039300-41 2023-10-20T03:36:00.774 INFO:tasks.rados.rados.0.smithi170.stdout:266: finishing write tid 6 to smithi17039300-41 2023-10-20T03:36:00.775 INFO:tasks.rados.rados.0.smithi170.stdout:266: oid 41 updating version 0 to 52 2023-10-20T03:36:00.775 INFO:tasks.rados.rados.0.smithi170.stdout:266: oid 41 version 52 is already newer than 50 2023-10-20T03:36:00.775 INFO:tasks.rados.rados.0.smithi170.stdout:266: oid 41 updating version 52 to 53 2023-10-20T03:36:00.775 INFO:tasks.rados.rados.0.smithi170.stdout:266: oid 41 updating version 53 to 54 2023-10-20T03:36:00.775 INFO:tasks.rados.rados.0.smithi170.stdout:266: oid 41 version 54 is already newer than 51 2023-10-20T03:36:00.776 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 41 v 54 (ObjNum 116 snap 22 seq_num 116) dirty exists 2023-10-20T03:36:00.776 INFO:tasks.rados.rados.0.smithi170.stdout:266: left oid 41 (ObjNum 116 snap 22 seq_num 116) 2023-10-20T03:36:00.776 INFO:tasks.rados.rados.0.smithi170.stdout:267: finishing write tid 1 to smithi17039300-36 2023-10-20T03:36:00.776 INFO:tasks.rados.rados.0.smithi170.stdout:267: finishing write tid 2 to smithi17039300-36 2023-10-20T03:36:00.776 INFO:tasks.rados.rados.0.smithi170.stdout:267: finishing write tid 3 to smithi17039300-36 2023-10-20T03:36:00.776 INFO:tasks.rados.rados.0.smithi170.stdout:267: finishing write tid 4 to smithi17039300-36 2023-10-20T03:36:00.777 INFO:tasks.rados.rados.0.smithi170.stdout:267: finishing write tid 5 to smithi17039300-36 2023-10-20T03:36:00.777 INFO:tasks.rados.rados.0.smithi170.stdout:267: finishing write tid 6 to smithi17039300-36 2023-10-20T03:36:00.777 INFO:tasks.rados.rados.0.smithi170.stdout:267: oid 36 updating version 0 to 43 2023-10-20T03:36:00.778 INFO:tasks.rados.rados.0.smithi170.stdout:267: oid 36 updating version 43 to 47 2023-10-20T03:36:00.778 INFO:tasks.rados.rados.0.smithi170.stdout:267: oid 36 version 47 is already newer than 44 2023-10-20T03:36:00.778 INFO:tasks.rados.rados.0.smithi170.stdout:267: oid 36 version 47 is already newer than 46 2023-10-20T03:36:00.778 INFO:tasks.rados.rados.0.smithi170.stdout:267: oid 36 version 47 is already newer than 45 2023-10-20T03:36:00.778 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 36 v 47 (ObjNum 117 snap 22 seq_num 117) dirty exists 2023-10-20T03:36:00.779 INFO:tasks.rados.rados.0.smithi170.stdout:267: left oid 36 (ObjNum 117 snap 22 seq_num 117) 2023-10-20T03:36:00.796 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:00.796 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:00.803 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:00.801+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:00.811 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-10-20T03:36:00.904 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:00.901+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:00.937 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:00.937 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:00.940 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:36:00.952 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-10-20T03:36:01.005 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:01.001+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:01.092 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:36:01.093 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.106 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:36:01.100+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:36:01.116 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:36:01.116 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.121 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:01.121 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:01.136 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-10-20T03:36:01.137 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:36:01.137 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.158 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:36:01.158 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.174 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:36:01.175 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.195 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:36:01.196 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.196 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:36:01.206 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:01.205+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:01.216 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:36:01.216 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.232 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:36:01.232 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.252 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:36:01.253 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.253 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:01.254 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:01.267 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-10-20T03:36:01.273 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:36:01.274 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.295 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:36:01.295 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.308 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:01.305+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:01.311 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:36:01.311 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.334 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:36:01.334 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.354 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:36:01.355 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.375 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:36:01.375 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:36:01.380 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:36:01.381 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:36:01.408 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:01.408 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:36:01.408 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:36:01.408 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:36:01.409 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:01.409 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:36:01.404+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:36:01.422 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-10-20T03:36:01.509 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:36:01.504+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:36:01.587 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:01.588 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:36:01.588 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:36:01.588 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:36:01.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.588 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:36:01.589 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:36:01.589 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 36.229220783000002, 2023-10-20T03:36:01.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010383547999999999, 2023-10-20T03:36:01.589 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.589 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:01.590 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:36:01.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.591 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.591 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:36:01.592 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.592 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.592 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.592 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.592 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356840+0000", 2023-10-20T03:36:01.592 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1999999999999999e-07 2023-10-20T03:36:01.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.593 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.593 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357506+0000", 2023-10-20T03:36:01.593 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00066620500000000001 2023-10-20T03:36:01.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.594 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.594 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.594 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357522+0000", 2023-10-20T03:36:01.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5964000000000002e-05 2023-10-20T03:36:01.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.595 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.595 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357535+0000", 2023-10-20T03:36:01.595 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3046e-05 2023-10-20T03:36:01.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.596 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.596 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.596 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357625+0000", 2023-10-20T03:36:01.596 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9691999999999994e-05 2023-10-20T03:36:01.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:01.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357636+0000", 2023-10-20T03:36:01.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1202e-05 2023-10-20T03:36:01.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.598 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:01.598 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.367176+0000", 2023-10-20T03:36:01.598 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0095396060000000008 2023-10-20T03:36:01.598 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.598 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.599 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.599 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.367223+0000", 2023-10-20T03:36:01.599 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7113000000000001e-05 2023-10-20T03:36:01.599 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.599 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.600 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.600 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=85, reqid=client.4570.0:312, at_version=43'25, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:36:01.600 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:36:01.601 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 36.211708385000001, 2023-10-20T03:36:01.601 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0052771249999999997, 2023-10-20T03:36:01.601 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.601 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:01.601 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.601 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.602 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.602 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:36:01.602 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.602 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.603 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.603 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:36:01.603 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.604 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.604 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374353+0000", 2023-10-20T03:36:01.604 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6199999999999999e-06 2023-10-20T03:36:01.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.604 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.605 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.605 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374496+0000", 2023-10-20T03:36:01.605 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014282800000000001 2023-10-20T03:36:01.605 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.605 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.605 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.606 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374515+0000", 2023-10-20T03:36:01.606 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8281e-05 2023-10-20T03:36:01.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.606 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374527+0000", 2023-10-20T03:36:01.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2835e-05 2023-10-20T03:36:01.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374607+0000", 2023-10-20T03:36:01.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9283999999999995e-05 2023-10-20T03:36:01.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374617+0000", 2023-10-20T03:36:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0394000000000001e-05 2023-10-20T03:36:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.379594+0000", 2023-10-20T03:36:01.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049767650000000002 2023-10-20T03:36:01.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.379629+0000", 2023-10-20T03:36:01.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5117999999999998e-05 2023-10-20T03:36:01.611 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.611 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.611 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.612 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.612 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=88, reqid=client.4570.0:313, at_version=43'26, trim_to=0'0, roll_forward_to=43'24))", 2023-10-20T03:36:01.612 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:36:01.612 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 36.198518083000003, 2023-10-20T03:36:01.612 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049208230000000004, 2023-10-20T03:36:01.613 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.613 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:01.613 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.613 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:36:01.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:36:01.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.615 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387544+0000", 2023-10-20T03:36:01.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6109999999999999e-06 2023-10-20T03:36:01.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387771+0000", 2023-10-20T03:36:01.617 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022766 2023-10-20T03:36:01.617 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.617 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.617 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.617 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387788+0000", 2023-10-20T03:36:01.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7079999999999999e-05 2023-10-20T03:36:01.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.618 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.618 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387799+0000", 2023-10-20T03:36:01.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0608000000000001e-05 2023-10-20T03:36:01.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387881+0000", 2023-10-20T03:36:01.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2060999999999999e-05 2023-10-20T03:36:01.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.620 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.620 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:01.620 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387892+0000", 2023-10-20T03:36:01.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0912e-05 2023-10-20T03:36:01.621 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.621 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.621 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:01.621 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.392420+0000", 2023-10-20T03:36:01.621 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0045283320000000004 2023-10-20T03:36:01.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.622 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.622 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.392463+0000", 2023-10-20T03:36:01.622 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2558999999999997e-05 2023-10-20T03:36:01.623 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.623 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.623 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.623 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.623 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:333 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~606208] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:01.624 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:36:01.624 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.200130311000002, 2023-10-20T03:36:01.624 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036441440999999998, 2023-10-20T03:36:01.624 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.624 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:01.625 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:01.625 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:01.625 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:01.625 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 333 2023-10-20T03:36:01.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.625 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.626 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.626 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:36:01.626 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.627 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.627 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:36:01.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385932+0000", 2023-10-20T03:36:01.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9929999999999998e-06 2023-10-20T03:36:01.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386889+0000", 2023-10-20T03:36:01.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000957437 2023-10-20T03:36:01.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.630 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.630 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386904+0000", 2023-10-20T03:36:01.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4669e-05 2023-10-20T03:36:01.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.631 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386910+0000", 2023-10-20T03:36:01.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3040000000000001e-06 2023-10-20T03:36:01.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.632 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386972+0000", 2023-10-20T03:36:01.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2003999999999998e-05 2023-10-20T03:36:01.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:01.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387043+0000", 2023-10-20T03:36:01.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0591000000000002e-05 2023-10-20T03:36:01.633 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.633 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:01.634 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388080+0000", 2023-10-20T03:36:01.634 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010373190000000001 2023-10-20T03:36:01.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.634 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:01.635 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393723+0000", 2023-10-20T03:36:01.635 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056426899999999997 2023-10-20T03:36:01.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.635 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.635 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:01.635 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422308+0000", 2023-10-20T03:36:01.636 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028584822999999999 2023-10-20T03:36:01.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.636 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.636 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.636 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422371+0000", 2023-10-20T03:36:01.636 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3610999999999998e-05 2023-10-20T03:36:01.637 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.637 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.637 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.638 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:334 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~720896] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:01.638 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:36:01.638 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.192995912000001, 2023-10-20T03:36:01.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029571147999999998, 2023-10-20T03:36:01.638 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.639 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:01.639 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:01.639 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:01.639 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:01.639 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 334 2023-10-20T03:36:01.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.640 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.640 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:36:01.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:36:01.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.642 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393066+0000", 2023-10-20T03:36:01.642 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8819999999999999e-06 2023-10-20T03:36:01.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.643 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393827+0000", 2023-10-20T03:36:01.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000760844 2023-10-20T03:36:01.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.644 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.644 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393842+0000", 2023-10-20T03:36:01.644 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5432999999999999e-05 2023-10-20T03:36:01.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393850+0000", 2023-10-20T03:36:01.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0199999999999994e-06 2023-10-20T03:36:01.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.646 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.646 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393915+0000", 2023-10-20T03:36:01.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4123999999999998e-05 2023-10-20T03:36:01.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.647 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:01.647 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394021+0000", 2023-10-20T03:36:01.647 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010668299999999999 2023-10-20T03:36:01.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.647 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.647 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:01.648 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394832+0000", 2023-10-20T03:36:01.648 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081066099999999996 2023-10-20T03:36:01.648 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.648 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.648 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:01.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405535+0000", 2023-10-20T03:36:01.649 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010703351 2023-10-20T03:36:01.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:01.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422553+0000", 2023-10-20T03:36:01.650 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017018021000000001 2023-10-20T03:36:01.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422635+0000", 2023-10-20T03:36:01.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2128999999999999e-05 2023-10-20T03:36:01.651 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.651 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.651 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.652 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:335 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~49152] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:01.652 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:36:01.652 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.192188936000001, 2023-10-20T03:36:01.652 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029832813999999999, 2023-10-20T03:36:01.653 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.653 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:01.653 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:01.653 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:01.653 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:01.654 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 335 2023-10-20T03:36:01.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.654 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.654 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.654 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:36:01.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.655 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.655 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:36:01.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393872+0000", 2023-10-20T03:36:01.657 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6000000000000003e-07 2023-10-20T03:36:01.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.657 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.657 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393919+0000", 2023-10-20T03:36:01.658 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6576000000000003e-05 2023-10-20T03:36:01.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.658 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.658 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393928+0000", 2023-10-20T03:36:01.659 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1220000000000001e-06 2023-10-20T03:36:01.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.659 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.659 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.659 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393937+0000", 2023-10-20T03:36:01.660 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1740000000000006e-06 2023-10-20T03:36:01.660 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.660 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.660 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.660 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401885+0000", 2023-10-20T03:36:01.661 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079486149999999992 2023-10-20T03:36:01.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.661 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:01.661 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.402012+0000", 2023-10-20T03:36:01.662 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012668 2023-10-20T03:36:01.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.662 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:01.662 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403063+0000", 2023-10-20T03:36:01.663 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010508410000000001 2023-10-20T03:36:01.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.663 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:01.663 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405538+0000", 2023-10-20T03:36:01.663 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024754519999999999 2023-10-20T03:36:01.664 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.664 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.664 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:01.664 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.423630+0000", 2023-10-20T03:36:01.664 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018091716000000001 2023-10-20T03:36:01.665 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.665 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.665 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.665 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.423704+0000", 2023-10-20T03:36:01.665 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3878e-05 2023-10-20T03:36:01.666 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.666 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.666 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.666 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.666 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.667 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:336 3.ds0 3:b8ff490e:::smithi17039300-34:head [setxattr _header (56) in=7b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:01.667 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:36:01.667 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.192109561000002, 2023-10-20T03:36:01.667 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030244758, 2023-10-20T03:36:01.667 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.667 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:01.668 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:01.668 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:01.668 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:01.668 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 336 2023-10-20T03:36:01.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.669 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.669 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.669 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:36:01.669 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.669 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.670 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.670 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.670 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:36:01.670 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.671 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.671 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.671 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:36:01.671 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8400000000000004e-07 2023-10-20T03:36:01.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.671 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.672 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.672 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393955+0000", 2023-10-20T03:36:01.672 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8210000000000003e-06 2023-10-20T03:36:01.672 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.672 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.672 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.673 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393963+0000", 2023-10-20T03:36:01.673 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8359999999999992e-06 2023-10-20T03:36:01.673 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.673 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.673 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393968+0000", 2023-10-20T03:36:01.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3689999999999998e-06 2023-10-20T03:36:01.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.674 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.675 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404110+0000", 2023-10-20T03:36:01.675 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010141582 2023-10-20T03:36:01.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:01.675 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404246+0000", 2023-10-20T03:36:01.676 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013631799999999999 2023-10-20T03:36:01.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.676 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:01.676 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405009+0000", 2023-10-20T03:36:01.677 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00076302800000000001 2023-10-20T03:36:01.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.677 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.677 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:01.677 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406198+0000", 2023-10-20T03:36:01.678 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011889610000000001 2023-10-20T03:36:01.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.678 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:01.678 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424049+0000", 2023-10-20T03:36:01.678 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017850957000000001 2023-10-20T03:36:01.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.679 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.679 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.679 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424195+0000", 2023-10-20T03:36:01.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014630200000000001 2023-10-20T03:36:01.680 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.680 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.680 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.680 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:337 3.ds0 3:b8ff490e:::smithi17039300-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:01.681 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:36:01.681 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.192082777000003, 2023-10-20T03:36:01.681 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.032645448000000001, 2023-10-20T03:36:01.681 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.681 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:36:01.681 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:01.682 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:01.682 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:01.682 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 337 2023-10-20T03:36:01.682 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.682 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.683 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.683 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.683 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:36:01.683 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.683 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.683 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:36:01.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.685 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.685 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393978+0000", 2023-10-20T03:36:01.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0800000000000005e-07 2023-10-20T03:36:01.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.685 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.685 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.686 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393981+0000", 2023-10-20T03:36:01.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7199999999999998e-06 2023-10-20T03:36:01.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393988+0000", 2023-10-20T03:36:01.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5939999999999996e-06 2023-10-20T03:36:01.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393993+0000", 2023-10-20T03:36:01.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0200000000000002e-06 2023-10-20T03:36:01.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.688 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.688 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405399+0000", 2023-10-20T03:36:01.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011406229 2023-10-20T03:36:01.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.689 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:01.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405522+0000", 2023-10-20T03:36:01.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012226199999999999 2023-10-20T03:36:01.690 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.690 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424833+0000", 2023-10-20T03:36:01.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019311524 2023-10-20T03:36:01.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.691 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.691 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:01.691 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424976+0000", 2023-10-20T03:36:01.691 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014297600000000001 2023-10-20T03:36:01.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.692 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.692 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.426623+0000", 2023-10-20T03:36:01.692 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0016466149999999999 2023-10-20T03:36:01.693 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.693 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.693 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.693 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.693 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:384 3.ds0 3:ba203010:::smithi17039300-4:head [copy-from ver 21] snapc f=[f,d,b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:36:01.694 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:36:01.694 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.215299495, 2023-10-20T03:36:01.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.077402023, 2023-10-20T03:36:01.694 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.694 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:01.695 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:01.695 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:01.695 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:01.695 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 384 2023-10-20T03:36:01.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.696 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:36:01.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:36:01.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370763+0000", 2023-10-20T03:36:01.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4569999999999999e-06 2023-10-20T03:36:01.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.699 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.699 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.699 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370776+0000", 2023-10-20T03:36:01.699 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2483000000000001e-05 2023-10-20T03:36:01.699 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.700 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.700 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370809+0000", 2023-10-20T03:36:01.700 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2916000000000001e-05 2023-10-20T03:36:01.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370826+0000", 2023-10-20T03:36:01.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7183000000000001e-05 2023-10-20T03:36:01.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.702 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.702 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371075+0000", 2023-10-20T03:36:01.702 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000249387 2023-10-20T03:36:01.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.702 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.703 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:01.703 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371389+0000", 2023-10-20T03:36:01.703 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031429899999999998 2023-10-20T03:36:01.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.703 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:01.704 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392222+0000", 2023-10-20T03:36:01.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020832532000000001 2023-10-20T03:36:01.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.704 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:01.705 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.447199+0000", 2023-10-20T03:36:01.705 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054976917 2023-10-20T03:36:01.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.705 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:01.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.448009+0000", 2023-10-20T03:36:01.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080978099999999998 2023-10-20T03:36:01.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.448163+0000", 2023-10-20T03:36:01.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015406799999999999 2023-10-20T03:36:01.707 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.707 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.707 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.708 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:385 3.ds0 3:ba203010:::smithi17039300-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:36:01.708 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:36:01.708 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.215199371000001, 2023-10-20T03:36:01.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.074370642000000001, 2023-10-20T03:36:01.708 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.709 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:36:01.709 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:01.709 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:01.709 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:01.709 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 385 2023-10-20T03:36:01.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.710 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.710 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.710 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:36:01.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.711 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.711 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:36:01.711 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.712 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.712 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.712 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370862+0000", 2023-10-20T03:36:01.712 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2699999999999998e-07 2023-10-20T03:36:01.712 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.712 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.713 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370866+0000", 2023-10-20T03:36:01.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.42e-06 2023-10-20T03:36:01.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370880+0000", 2023-10-20T03:36:01.714 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3833e-05 2023-10-20T03:36:01.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.715 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370889+0000", 2023-10-20T03:36:01.715 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5900000000000008e-06 2023-10-20T03:36:01.715 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.715 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.716 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371887+0000", 2023-10-20T03:36:01.716 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00099814499999999998 2023-10-20T03:36:01.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.716 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.716 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2023-10-20T03:36:01.716 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.372069+0000", 2023-10-20T03:36:01.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018220899999999999 2023-10-20T03:36:01.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.717 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.717 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445044+0000", 2023-10-20T03:36:01.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072974686999999996 2023-10-20T03:36:01.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.718 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:01.718 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445137+0000", 2023-10-20T03:36:01.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3694000000000001e-05 2023-10-20T03:36:01.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.719 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.719 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445231+0000", 2023-10-20T03:36:01.719 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4136999999999997e-05 2023-10-20T03:36:01.719 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.720 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.720 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.720 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(osd.12.22:4 3.ds0 3:b8ff490e:::smithi17039300-34:head [assert-version v21,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:36:01.721 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:36:01.721 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.212750648, 2023-10-20T03:36:01.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083526739999999992, 2023-10-20T03:36:01.721 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.721 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:36:01.721 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.722 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.722 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.722 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:36:01.722 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.723 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.723 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.723 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:36:01.723 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.723 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.723 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.724 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.724 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373312+0000", 2023-10-20T03:36:01.724 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.424e-06 2023-10-20T03:36:01.724 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.724 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.725 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.725 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373326+0000", 2023-10-20T03:36:01.725 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4226000000000001e-05 2023-10-20T03:36:01.725 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.725 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.725 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.726 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373357+0000", 2023-10-20T03:36:01.726 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0698999999999999e-05 2023-10-20T03:36:01.726 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.726 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.726 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.726 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373388+0000", 2023-10-20T03:36:01.727 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0846000000000002e-05 2023-10-20T03:36:01.727 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.727 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.727 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.727 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373551+0000", 2023-10-20T03:36:01.727 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016350300000000001 2023-10-20T03:36:01.728 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.728 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.728 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:01.728 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373703+0000", 2023-10-20T03:36:01.728 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015216200000000001 2023-10-20T03:36:01.729 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.729 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.729 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.729 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.381662+0000", 2023-10-20T03:36:01.729 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079588139999999998 2023-10-20T03:36:01.729 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.730 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.730 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.730 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.730 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.730 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpReadReply(3.ds0 48/31 ECSubReadReply(tid=59, attrs_read=0))", 2023-10-20T03:36:01.731 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:36:01.731 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.209818129999999, 2023-10-20T03:36:01.731 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0055763890000000002, 2023-10-20T03:36:01.731 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.731 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:01.732 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.732 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.732 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.732 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:36:01.732 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.733 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.733 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.733 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.733 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:36:01.733 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.733 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.734 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.734 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.734 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376244+0000", 2023-10-20T03:36:01.734 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7570000000000001e-06 2023-10-20T03:36:01.734 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.735 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.735 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.735 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380643+0000", 2023-10-20T03:36:01.735 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00439931 2023-10-20T03:36:01.735 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.736 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.736 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.736 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380671+0000", 2023-10-20T03:36:01.736 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8019000000000001e-05 2023-10-20T03:36:01.736 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.736 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.737 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.737 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380685+0000", 2023-10-20T03:36:01.737 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4334e-05 2023-10-20T03:36:01.737 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.737 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.737 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.738 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380780+0000", 2023-10-20T03:36:01.738 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4087999999999998e-05 2023-10-20T03:36:01.738 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.738 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.738 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.739 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.381818+0000", 2023-10-20T03:36:01.739 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001038881 2023-10-20T03:36:01.739 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.739 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.739 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.740 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.740 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.740 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs2 48/47 ECSubWrite(tid=115, reqid=osd.13.0:115, at_version=48'27, trim_to=0'0, roll_forward_to=0'0))", 2023-10-20T03:36:01.740 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:36:01.740 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.196520258, 2023-10-20T03:36:01.741 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047760049999999998, 2023-10-20T03:36:01.741 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.741 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:01.741 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.741 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.741 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.742 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:36:01.742 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.742 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.742 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.742 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.743 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:36:01.743 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.743 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.743 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.743 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.744 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389541+0000", 2023-10-20T03:36:01.744 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5659999999999999e-06 2023-10-20T03:36:01.744 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.744 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.744 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.745 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389579+0000", 2023-10-20T03:36:01.745 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7070999999999998e-05 2023-10-20T03:36:01.745 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.745 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.745 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.745 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389605+0000", 2023-10-20T03:36:01.746 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6848000000000001e-05 2023-10-20T03:36:01.746 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.746 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.746 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.746 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389626+0000", 2023-10-20T03:36:01.747 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0579000000000001e-05 2023-10-20T03:36:01.747 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.747 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.747 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.747 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389781+0000", 2023-10-20T03:36:01.748 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015502300000000001 2023-10-20T03:36:01.748 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.748 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.748 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:01.748 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389807+0000", 2023-10-20T03:36:01.748 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6418999999999999e-05 2023-10-20T03:36:01.749 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.749 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.749 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:01.749 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394222+0000", 2023-10-20T03:36:01.749 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044143139999999999 2023-10-20T03:36:01.750 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.750 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.750 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.750 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394316+0000", 2023-10-20T03:36:01.750 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4184999999999994e-05 2023-10-20T03:36:01.750 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.751 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.751 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.751 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.751 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.751 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:01.752 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:36:01.752 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.156645145999999, 2023-10-20T03:36:01.752 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015918188, 2023-10-20T03:36:01.752 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.752 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:01.753 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.753 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.753 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.753 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:36:01.753 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.754 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.754 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.754 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.754 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:36:01.754 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.755 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.755 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.755 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.755 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429417+0000", 2023-10-20T03:36:01.755 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9429999999999999e-06 2023-10-20T03:36:01.755 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.756 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.756 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.756 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429432+0000", 2023-10-20T03:36:01.756 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4942e-05 2023-10-20T03:36:01.756 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.756 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.757 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.757 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429459+0000", 2023-10-20T03:36:01.757 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6740000000000001e-05 2023-10-20T03:36:01.757 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.757 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.758 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.758 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429472+0000", 2023-10-20T03:36:01.758 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3382e-05 2023-10-20T03:36:01.758 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.758 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.758 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.759 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445294+0000", 2023-10-20T03:36:01.759 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015821894999999999 2023-10-20T03:36:01.759 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.759 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.759 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.759 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445333+0000", 2023-10-20T03:36:01.760 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9286000000000003e-05 2023-10-20T03:36:01.760 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.760 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.760 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.760 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.760 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.761 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:01.761 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:36:01.761 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.156296163, 2023-10-20T03:36:01.761 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01562377, 2023-10-20T03:36:01.761 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.761 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:01.762 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.762 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.762 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.762 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:36:01.762 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.763 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.763 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.763 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.763 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:36:01.763 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.763 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.764 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.764 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.764 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429765+0000", 2023-10-20T03:36:01.764 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0079999999999999e-06 2023-10-20T03:36:01.764 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.764 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.765 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.765 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429772+0000", 2023-10-20T03:36:01.765 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2230000000000001e-06 2023-10-20T03:36:01.765 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.765 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.766 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.766 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429793+0000", 2023-10-20T03:36:01.766 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0707999999999999e-05 2023-10-20T03:36:01.766 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.766 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.767 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.767 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429806+0000", 2023-10-20T03:36:01.767 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3528e-05 2023-10-20T03:36:01.767 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.767 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.767 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.768 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445370+0000", 2023-10-20T03:36:01.768 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015563917 2023-10-20T03:36:01.768 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.768 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.768 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.769 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445388+0000", 2023-10-20T03:36:01.769 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7385999999999999e-05 2023-10-20T03:36:01.769 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.769 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.769 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.770 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.770 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.770 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:01.770 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:36:01.770 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.154645266999999, 2023-10-20T03:36:01.771 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014045581999999999, 2023-10-20T03:36:01.771 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.771 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:01.771 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.771 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.772 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.772 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:36:01.772 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.772 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.772 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.772 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.773 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:36:01.773 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.773 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.773 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.773 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.774 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431417+0000", 2023-10-20T03:36:01.774 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4159999999999998e-06 2023-10-20T03:36:01.774 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.774 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.774 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.775 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431433+0000", 2023-10-20T03:36:01.775 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5653000000000001e-05 2023-10-20T03:36:01.775 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.775 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.775 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.775 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431472+0000", 2023-10-20T03:36:01.776 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9232e-05 2023-10-20T03:36:01.776 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.776 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.776 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.776 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431492+0000", 2023-10-20T03:36:01.777 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9550000000000001e-05 2023-10-20T03:36:01.777 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.777 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.777 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.777 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445435+0000", 2023-10-20T03:36:01.778 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01394307 2023-10-20T03:36:01.778 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.778 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.778 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.778 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445460+0000", 2023-10-20T03:36:01.778 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5661e-05 2023-10-20T03:36:01.779 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.779 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.779 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.779 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.779 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.780 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:01.780 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:36:01.780 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.153839266999999, 2023-10-20T03:36:01.780 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013263258999999999, 2023-10-20T03:36:01.780 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.781 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:01.781 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.781 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.781 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.781 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:36:01.781 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.782 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.782 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.782 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.782 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:36:01.782 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.783 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.783 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.783 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432228+0000", 2023-10-20T03:36:01.783 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.08e-06 2023-10-20T03:36:01.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.784 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.784 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.784 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432236+0000", 2023-10-20T03:36:01.784 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6629999999999993e-06 2023-10-20T03:36:01.784 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.785 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.785 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432260+0000", 2023-10-20T03:36:01.785 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3998999999999998e-05 2023-10-20T03:36:01.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.786 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.786 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432272+0000", 2023-10-20T03:36:01.786 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2023e-05 2023-10-20T03:36:01.786 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.786 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.786 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.787 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445470+0000", 2023-10-20T03:36:01.787 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013198507 2023-10-20T03:36:01.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.787 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.787 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.788 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445484+0000", 2023-10-20T03:36:01.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3987e-05 2023-10-20T03:36:01.788 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.788 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.788 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.789 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.789 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:01.789 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:36:01.789 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 31.153534917999998, 2023-10-20T03:36:01.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012997794, 2023-10-20T03:36:01.790 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.790 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:01.790 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.790 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.791 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:36:01.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.791 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.791 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:36:01.792 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.792 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.792 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432526+0000", 2023-10-20T03:36:01.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1400000000000001e-06 2023-10-20T03:36:01.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.793 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.793 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432532+0000", 2023-10-20T03:36:01.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.823e-06 2023-10-20T03:36:01.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432552+0000", 2023-10-20T03:36:01.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0032999999999999e-05 2023-10-20T03:36:01.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432567+0000", 2023-10-20T03:36:01.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4932999999999999e-05 2023-10-20T03:36:01.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.796 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445506+0000", 2023-10-20T03:36:01.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012939315 2023-10-20T03:36:01.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.797 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.797 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445523+0000", 2023-10-20T03:36:01.797 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6549999999999999e-05 2023-10-20T03:36:01.797 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.798 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.798 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.798 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.798 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.798 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs2 49/47 ECSubWrite(tid=118, reqid=osd.13.0:118, at_version=49'29, trim_to=0'0, roll_forward_to=48'27))", 2023-10-20T03:36:01.799 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:36:01.799 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 30.175201687000001, 2023-10-20T03:36:01.799 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043378410000000003, 2023-10-20T03:36:01.799 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.799 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:01.799 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.800 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.800 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.800 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:36:01.800 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.800 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.801 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.801 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:36:01.801 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.802 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.802 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.802 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410859+0000", 2023-10-20T03:36:01.802 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0699999999999999e-07 2023-10-20T03:36:01.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.803 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.803 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410884+0000", 2023-10-20T03:36:01.803 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.529e-05 2023-10-20T03:36:01.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.804 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.804 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.804 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410898+0000", 2023-10-20T03:36:01.804 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.383e-05 2023-10-20T03:36:01.804 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.805 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410928+0000", 2023-10-20T03:36:01.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0329e-05 2023-10-20T03:36:01.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.806 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.806 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.411037+0000", 2023-10-20T03:36:01.806 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010869099999999999 2023-10-20T03:36:01.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:01.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.411048+0000", 2023-10-20T03:36:01.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0905e-05 2023-10-20T03:36:01.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.808 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:01.808 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415159+0000", 2023-10-20T03:36:01.808 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041115520000000001 2023-10-20T03:36:01.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.809 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.809 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415196+0000", 2023-10-20T03:36:01.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7036999999999998e-05 2023-10-20T03:36:01.809 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.809 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.809 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.810 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpRead(3.cs2 55/54 ECSubRead(tid=70, to_read={3:3382ea17:::smithi17039300-35:6=0,262144,0}, subchunks={3:3382ea17:::smithi17039300-35:6=[0,1]}, attrs_to_read=))", 2023-10-20T03:36:01.810 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:37.459240+0000", 2023-10-20T03:36:01.810 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.126820581, 2023-10-20T03:36:01.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044608460000000001, 2023-10-20T03:36:01.811 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:01.811 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:01.811 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:01.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:01.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459240+0000", 2023-10-20T03:36:01.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:01.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459240+0000", 2023-10-20T03:36:01.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:01.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:01.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459241+0000", 2023-10-20T03:36:01.814 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.84e-06 2023-10-20T03:36:01.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.814 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:01.814 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459256+0000", 2023-10-20T03:36:01.815 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.454e-05 2023-10-20T03:36:01.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.815 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.815 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:01.815 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459288+0000", 2023-10-20T03:36:01.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2329000000000002e-05 2023-10-20T03:36:01.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.816 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:01.816 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459307+0000", 2023-10-20T03:36:01.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9046999999999999e-05 2023-10-20T03:36:01.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.817 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:01.817 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459460+0000", 2023-10-20T03:36:01.817 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015253200000000001 2023-10-20T03:36:01.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:01.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:01.818 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:01.818 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.463700+0000", 2023-10-20T03:36:01.818 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042405580000000002 2023-10-20T03:36:01.819 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.819 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.819 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.819 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:01.819 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:01.819 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:01.820 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-10-20T03:36:01.821 INFO:tasks.rados.rados.0.smithi170.stdout:266: done (2 left) 2023-10-20T03:36:01.821 INFO:tasks.rados.rados.0.smithi170.stdout:267: done (1 left) 2023-10-20T03:36:01.822 INFO:tasks.rados.rados.0.smithi170.stdout:269: done (0 left) 2023-10-20T03:36:01.822 INFO:tasks.rados.rados.0.smithi170.stdout:270: snap_remove snap 22 2023-10-20T03:36:01.822 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:01.821+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:01.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-10-20T03:36:01.922 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:01.921+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:01.959 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:01.960 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:36:01.960 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:36:01.960 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:01.974 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-10-20T03:36:02.022 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:02.020+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:02.101 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:36:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:36:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:36:02.102 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:02.115 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-10-20T03:36:02.123 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:02.120+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:02.224 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:02.220+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:02.288 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:02.288 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:36:02.288 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:36:02.288 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2023-10-20T03:36:02.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.289 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.bs1 41/31 ECSubWrite(tid=77, reqid=client.4570.0:293, at_version=41'10, trim_to=0'0, roll_forward_to=39'8))", 2023-10-20T03:36:02.289 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:23.461353+0000", 2023-10-20T03:36:02.289 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 38.824877825999998, 2023-10-20T03:36:02.289 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018876751000000001, 2023-10-20T03:36:02.290 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.290 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:02.290 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.290 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.291 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.461353+0000", 2023-10-20T03:36:02.291 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.291 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.461353+0000", 2023-10-20T03:36:02.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.293 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.461354+0000", 2023-10-20T03:36:02.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.029e-06 2023-10-20T03:36:02.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462244+0000", 2023-10-20T03:36:02.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089060400000000003 2023-10-20T03:36:02.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.295 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462291+0000", 2023-10-20T03:36:02.295 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6476e-05 2023-10-20T03:36:02.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.295 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.295 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462316+0000", 2023-10-20T03:36:02.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5320000000000002e-05 2023-10-20T03:36:02.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462615+0000", 2023-10-20T03:36:02.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00029899800000000003 2023-10-20T03:36:02.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:02.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462645+0000", 2023-10-20T03:36:02.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0352999999999999e-05 2023-10-20T03:36:02.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.298 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:02.298 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.480122+0000", 2023-10-20T03:36:02.299 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017476588000000001 2023-10-20T03:36:02.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.299 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.299 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.480229+0000", 2023-10-20T03:36:02.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107383 2023-10-20T03:36:02.300 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.300 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.300 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.301 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.fs6 53/47 ECSubWrite(tid=57, reqid=client.4570.0:408, at_version=53'31, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:02.301 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:35.448443+0000", 2023-10-20T03:36:02.301 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.837787811999998, 2023-10-20T03:36:02.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015353739, 2023-10-20T03:36:02.302 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.302 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:02.302 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.448443+0000", 2023-10-20T03:36:02.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.303 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.448443+0000", 2023-10-20T03:36:02.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.448445+0000", 2023-10-20T03:36:02.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3999999999999999e-06 2023-10-20T03:36:02.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.449247+0000", 2023-10-20T03:36:02.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00080198499999999996 2023-10-20T03:36:02.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.449274+0000", 2023-10-20T03:36:02.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7336999999999999e-05 2023-10-20T03:36:02.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.449289+0000", 2023-10-20T03:36:02.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4222999999999999e-05 2023-10-20T03:36:02.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.449396+0000", 2023-10-20T03:36:02.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107756 2023-10-20T03:36:02.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:02.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.449416+0000", 2023-10-20T03:36:02.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9606000000000002e-05 2023-10-20T03:36:02.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:02.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.463712+0000", 2023-10-20T03:36:02.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014295809 2023-10-20T03:36:02.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.311 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.463796+0000", 2023-10-20T03:36:02.312 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4623000000000006e-05 2023-10-20T03:36:02.312 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.312 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.312 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.313 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.fs6 53/47 ECSubWrite(tid=58, reqid=client.4570.0:409, at_version=53'32, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:02.313 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:35.472532+0000", 2023-10-20T03:36:02.313 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.813698511999998, 2023-10-20T03:36:02.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016132318, 2023-10-20T03:36:02.313 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.314 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:02.314 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.472532+0000", 2023-10-20T03:36:02.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.472532+0000", 2023-10-20T03:36:02.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.316 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.472534+0000", 2023-10-20T03:36:02.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.643e-06 2023-10-20T03:36:02.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.317 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.472866+0000", 2023-10-20T03:36:02.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00033227400000000002 2023-10-20T03:36:02.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.472897+0000", 2023-10-20T03:36:02.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1106999999999999e-05 2023-10-20T03:36:02.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.472912+0000", 2023-10-20T03:36:02.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5322999999999999e-05 2023-10-20T03:36:02.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.473029+0000", 2023-10-20T03:36:02.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011634299999999999 2023-10-20T03:36:02.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:02.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.473046+0000", 2023-10-20T03:36:02.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7546000000000001e-05 2023-10-20T03:36:02.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:02.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.488610+0000", 2023-10-20T03:36:02.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015563926 2023-10-20T03:36:02.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.488664+0000", 2023-10-20T03:36:02.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4156000000000001e-05 2023-10-20T03:36:02.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.324 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.325 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.fs6 60/47 ECSubWrite(tid=88, reqid=osd.3.0:87, at_version=0'0, trim_to=0'0, roll_forward_to=60'54))", 2023-10-20T03:36:02.325 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.899284+0000", 2023-10-20T03:36:02.325 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.386946642000002, 2023-10-20T03:36:02.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016204105999999999, 2023-10-20T03:36:02.325 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.326 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:02.326 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.326 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.326 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899284+0000", 2023-10-20T03:36:02.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899284+0000", 2023-10-20T03:36:02.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899285+0000", 2023-10-20T03:36:02.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.404e-06 2023-10-20T03:36:02.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899306+0000", 2023-10-20T03:36:02.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0766000000000001e-05 2023-10-20T03:36:02.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899326+0000", 2023-10-20T03:36:02.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0190000000000002e-05 2023-10-20T03:36:02.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.331 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.331 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899335+0000", 2023-10-20T03:36:02.331 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2099999999999999e-06 2023-10-20T03:36:02.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899429+0000", 2023-10-20T03:36:02.332 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3954000000000002e-05 2023-10-20T03:36:02.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:02.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899441+0000", 2023-10-20T03:36:02.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1127e-05 2023-10-20T03:36:02.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:02.334 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.915423+0000", 2023-10-20T03:36:02.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015982845999999998 2023-10-20T03:36:02.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.915488+0000", 2023-10-20T03:36:02.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4609000000000004e-05 2023-10-20T03:36:02.335 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.336 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.336 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.336 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.7s3 60/31 ECSubWrite(tid=200, reqid=client.4570.0:459, at_version=60'9, trim_to=0'0, roll_forward_to=38'7))", 2023-10-20T03:36:02.337 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.899303+0000", 2023-10-20T03:36:02.337 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.386927136000001, 2023-10-20T03:36:02.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.023677786999999999, 2023-10-20T03:36:02.337 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.337 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:02.338 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899303+0000", 2023-10-20T03:36:02.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.339 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899303+0000", 2023-10-20T03:36:02.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899305+0000", 2023-10-20T03:36:02.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.813e-06 2023-10-20T03:36:02.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899602+0000", 2023-10-20T03:36:02.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00029634300000000002 2023-10-20T03:36:02.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899630+0000", 2023-10-20T03:36:02.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8085999999999998e-05 2023-10-20T03:36:02.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.343 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.343 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899663+0000", 2023-10-20T03:36:02.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3648e-05 2023-10-20T03:36:02.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899783+0000", 2023-10-20T03:36:02.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119461 2023-10-20T03:36:02.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:02.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899803+0000", 2023-10-20T03:36:02.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0335000000000001e-05 2023-10-20T03:36:02.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:02.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.922864+0000", 2023-10-20T03:36:02.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.023061324000000001 2023-10-20T03:36:02.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.922981+0000", 2023-10-20T03:36:02.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000116777 2023-10-20T03:36:02.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.347 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.348 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.7s3 60/31 ECSubWrite(tid=203, reqid=client.4570.0:460, at_version=60'10, trim_to=0'0, roll_forward_to=38'7))", 2023-10-20T03:36:02.348 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.916054+0000", 2023-10-20T03:36:02.349 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.370176565000001, 2023-10-20T03:36:02.349 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014636858000000001, 2023-10-20T03:36:02.349 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.349 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:02.349 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.916054+0000", 2023-10-20T03:36:02.350 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.351 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.351 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.916054+0000", 2023-10-20T03:36:02.351 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.352 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.352 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.916055+0000", 2023-10-20T03:36:02.352 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.469e-06 2023-10-20T03:36:02.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.353 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.916256+0000", 2023-10-20T03:36:02.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020046699999999999 2023-10-20T03:36:02.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.916277+0000", 2023-10-20T03:36:02.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1364000000000001e-05 2023-10-20T03:36:02.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.916291+0000", 2023-10-20T03:36:02.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4188e-05 2023-10-20T03:36:02.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.356 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.356 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.916394+0000", 2023-10-20T03:36:02.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000102159 2023-10-20T03:36:02.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.356 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:02.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.916412+0000", 2023-10-20T03:36:02.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8099000000000001e-05 2023-10-20T03:36:02.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:02.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.930630+0000", 2023-10-20T03:36:02.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014218798 2023-10-20T03:36:02.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.930691+0000", 2023-10-20T03:36:02.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0313999999999999e-05 2023-10-20T03:36:02.359 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.359 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.359 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.360 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:545 3.8s1 3:1166f250:::smithi17039300-32:head [read 0~3375104,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2023-10-20T03:36:02.360 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:46.883962+0000", 2023-10-20T03:36:02.360 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 15.402268259, 2023-10-20T03:36:02.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025717769000000001, 2023-10-20T03:36:02.361 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.361 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:02.361 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:02.361 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:02.362 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:02.362 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 545 2023-10-20T03:36:02.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.362 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.362 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.883962+0000", 2023-10-20T03:36:02.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.883962+0000", 2023-10-20T03:36:02.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.883967+0000", 2023-10-20T03:36:02.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.284e-06 2023-10-20T03:36:02.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.883991+0000", 2023-10-20T03:36:02.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4093000000000001e-05 2023-10-20T03:36:02.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.366 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884042+0000", 2023-10-20T03:36:02.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1819e-05 2023-10-20T03:36:02.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884070+0000", 2023-10-20T03:36:02.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.758e-05 2023-10-20T03:36:02.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884323+0000", 2023-10-20T03:36:02.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025264500000000002 2023-10-20T03:36:02.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op must wait for map", 2023-10-20T03:36:02.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884348+0000", 2023-10-20T03:36:02.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5276e-05 2023-10-20T03:36:02.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.370 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.889737+0000", 2023-10-20T03:36:02.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053888989999999999 2023-10-20T03:36:02.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:02.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.890003+0000", 2023-10-20T03:36:02.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026633800000000002 2023-10-20T03:36:02.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.909680+0000", 2023-10-20T03:36:02.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019676835 2023-10-20T03:36:02.372 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.372 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.373 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.373 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:546 3.8s1 3:1166f250:::smithi17039300-32:head [read 0~3375104,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2023-10-20T03:36:02.373 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:46.884128+0000", 2023-10-20T03:36:02.374 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 15.402102345999999, 2023-10-20T03:36:02.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031700702999999997, 2023-10-20T03:36:02.374 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.374 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:02.374 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:02.375 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:02.375 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:02.375 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 546 2023-10-20T03:36:02.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.376 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884128+0000", 2023-10-20T03:36:02.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884128+0000", 2023-10-20T03:36:02.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884130+0000", 2023-10-20T03:36:02.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.536e-06 2023-10-20T03:36:02.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884140+0000", 2023-10-20T03:36:02.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0186999999999999e-05 2023-10-20T03:36:02.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884165+0000", 2023-10-20T03:36:02.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5276e-05 2023-10-20T03:36:02.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884183+0000", 2023-10-20T03:36:02.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7496999999999999e-05 2023-10-20T03:36:02.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884489+0000", 2023-10-20T03:36:02.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030652099999999997 2023-10-20T03:36:02.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:02.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884510+0000", 2023-10-20T03:36:02.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.088e-05 2023-10-20T03:36:02.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.895027+0000", 2023-10-20T03:36:02.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010516635999999999 2023-10-20T03:36:02.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:02.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.895341+0000", 2023-10-20T03:36:02.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00031437799999999998 2023-10-20T03:36:02.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.915829+0000", 2023-10-20T03:36:02.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020487792000000001 2023-10-20T03:36:02.386 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.386 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.386 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.387 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:547 3.8s1 3:1166f250:::smithi17039300-32:head [read 0~3375104,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2023-10-20T03:36:02.387 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:46.884211+0000", 2023-10-20T03:36:02.387 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 15.402019600999999, 2023-10-20T03:36:02.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.037539995999999999, 2023-10-20T03:36:02.388 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.388 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:02.388 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:02.388 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:02.388 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:02.389 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 547 2023-10-20T03:36:02.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.389 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884211+0000", 2023-10-20T03:36:02.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884211+0000", 2023-10-20T03:36:02.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884212+0000", 2023-10-20T03:36:02.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0669999999999999e-06 2023-10-20T03:36:02.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884218+0000", 2023-10-20T03:36:02.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9699999999999996e-06 2023-10-20T03:36:02.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884262+0000", 2023-10-20T03:36:02.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3949999999999998e-05 2023-10-20T03:36:02.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884280+0000", 2023-10-20T03:36:02.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7632999999999999e-05 2023-10-20T03:36:02.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884611+0000", 2023-10-20T03:36:02.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00033177400000000001 2023-10-20T03:36:02.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:02.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884634+0000", 2023-10-20T03:36:02.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2909999999999999e-05 2023-10-20T03:36:02.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.897834+0000", 2023-10-20T03:36:02.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013200197 2023-10-20T03:36:02.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:02.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.898033+0000", 2023-10-20T03:36:02.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019837500000000001 2023-10-20T03:36:02.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.921751+0000", 2023-10-20T03:36:02.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.023718119999999999 2023-10-20T03:36:02.399 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.399 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.400 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.400 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpReadReply(3.8s1 64/31 ECSubReadReply(tid=49, attrs_read=0))", 2023-10-20T03:36:02.400 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:46.899743+0000", 2023-10-20T03:36:02.401 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 15.386487238000001, 2023-10-20T03:36:02.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016233357, 2023-10-20T03:36:02.401 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.401 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:02.401 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.899743+0000", 2023-10-20T03:36:02.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.899743+0000", 2023-10-20T03:36:02.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.899744+0000", 2023-10-20T03:36:02.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5300000000000003e-07 2023-10-20T03:36:02.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.903052+0000", 2023-10-20T03:36:02.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0033080090000000002 2023-10-20T03:36:02.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.903086+0000", 2023-10-20T03:36:02.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3531999999999997e-05 2023-10-20T03:36:02.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.903128+0000", 2023-10-20T03:36:02.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2033999999999998e-05 2023-10-20T03:36:02.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.910545+0000", 2023-10-20T03:36:02.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0074171239999999998 2023-10-20T03:36:02.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.915977+0000", 2023-10-20T03:36:02.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0054318049999999996 2023-10-20T03:36:02.409 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.409 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.409 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.410 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpReadReply(3.8s1 65/31 ECSubReadReply(tid=50, attrs_read=0))", 2023-10-20T03:36:02.410 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:46.900134+0000", 2023-10-20T03:36:02.410 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 15.386096315, 2023-10-20T03:36:02.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016023096000000001, 2023-10-20T03:36:02.411 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.411 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:02.411 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.900134+0000", 2023-10-20T03:36:02.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.900134+0000", 2023-10-20T03:36:02.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.900135+0000", 2023-10-20T03:36:02.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6299999999999993e-07 2023-10-20T03:36:02.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.902820+0000", 2023-10-20T03:36:02.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026850950000000002 2023-10-20T03:36:02.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.902836+0000", 2023-10-20T03:36:02.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5489999999999999e-05 2023-10-20T03:36:02.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.902846+0000", 2023-10-20T03:36:02.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0056e-05 2023-10-20T03:36:02.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.916078+0000", 2023-10-20T03:36:02.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013232036000000001 2023-10-20T03:36:02.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.916157+0000", 2023-10-20T03:36:02.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9456000000000006e-05 2023-10-20T03:36:02.419 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.419 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.419 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.420 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpReadReply(3.8s1 65/31 ECSubReadReply(tid=50, attrs_read=0))", 2023-10-20T03:36:02.420 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:46.900716+0000", 2023-10-20T03:36:02.420 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 15.385515003, 2023-10-20T03:36:02.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015721077999999999, 2023-10-20T03:36:02.420 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.421 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:02.421 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.900716+0000", 2023-10-20T03:36:02.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.900716+0000", 2023-10-20T03:36:02.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.900716+0000", 2023-10-20T03:36:02.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3e-07 2023-10-20T03:36:02.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.902090+0000", 2023-10-20T03:36:02.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001374077 2023-10-20T03:36:02.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.902106+0000", 2023-10-20T03:36:02.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5614e-05 2023-10-20T03:36:02.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.902115+0000", 2023-10-20T03:36:02.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4539999999999996e-06 2023-10-20T03:36:02.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.916380+0000", 2023-10-20T03:36:02.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014264796999999999 2023-10-20T03:36:02.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.916437+0000", 2023-10-20T03:36:02.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6406000000000002e-05 2023-10-20T03:36:02.428 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.428 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.428 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.429 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpReadReply(3.8s1 65/31 ECSubReadReply(tid=50, attrs_read=0))", 2023-10-20T03:36:02.429 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:46.903170+0000", 2023-10-20T03:36:02.429 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 15.383060177000001, 2023-10-20T03:36:02.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018730475, 2023-10-20T03:36:02.430 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.430 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:02.430 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.903170+0000", 2023-10-20T03:36:02.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.903170+0000", 2023-10-20T03:36:02.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.903171+0000", 2023-10-20T03:36:02.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.113e-06 2023-10-20T03:36:02.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.904062+0000", 2023-10-20T03:36:02.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089037999999999997 2023-10-20T03:36:02.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.904081+0000", 2023-10-20T03:36:02.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8766999999999998e-05 2023-10-20T03:36:02.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.904091+0000", 2023-10-20T03:36:02.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813e-05 2023-10-20T03:36:02.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.916506+0000", 2023-10-20T03:36:02.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012414236 2023-10-20T03:36:02.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.921901+0000", 2023-10-20T03:36:02.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053951659999999999 2023-10-20T03:36:02.439 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.439 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.439 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.440 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.3s2 70/56 ECSubWrite(tid=55, reqid=client.4570.0:556, at_version=70'40, trim_to=0'0, roll_forward_to=60'38))", 2023-10-20T03:36:02.440 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:51.966440+0000", 2023-10-20T03:36:02.440 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.319790151999999, 2023-10-20T03:36:02.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015551456, 2023-10-20T03:36:02.440 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.441 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:02.441 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:51.966440+0000", 2023-10-20T03:36:02.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:51.966440+0000", 2023-10-20T03:36:02.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:51.966441+0000", 2023-10-20T03:36:02.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.37e-07 2023-10-20T03:36:02.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:51.967059+0000", 2023-10-20T03:36:02.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00061826899999999996 2023-10-20T03:36:02.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:51.967096+0000", 2023-10-20T03:36:02.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6195999999999997e-05 2023-10-20T03:36:02.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:51.967120+0000", 2023-10-20T03:36:02.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4700000000000001e-05 2023-10-20T03:36:02.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:51.967308+0000", 2023-10-20T03:36:02.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018732500000000001 2023-10-20T03:36:02.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:02.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:51.967341+0000", 2023-10-20T03:36:02.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3359999999999999e-05 2023-10-20T03:36:02.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:02.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:51.981916+0000", 2023-10-20T03:36:02.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014574755999999999 2023-10-20T03:36:02.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:51.981992+0000", 2023-10-20T03:36:02.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6112999999999994e-05 2023-10-20T03:36:02.452 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.452 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.453 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.453 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.3s2 71/56 ECSubWrite(tid=63, reqid=client.4570.0:575, at_version=71'46, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:02.453 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:52.983764+0000", 2023-10-20T03:36:02.454 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 9.3024663889999992, 2023-10-20T03:36:02.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018745662, 2023-10-20T03:36:02.454 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.454 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:02.454 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.983764+0000", 2023-10-20T03:36:02.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.983764+0000", 2023-10-20T03:36:02.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.983766+0000", 2023-10-20T03:36:02.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.657e-06 2023-10-20T03:36:02.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.984152+0000", 2023-10-20T03:36:02.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038608000000000002 2023-10-20T03:36:02.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.984182+0000", 2023-10-20T03:36:02.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0360000000000001e-05 2023-10-20T03:36:02.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.984198+0000", 2023-10-20T03:36:02.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5290000000000001e-05 2023-10-20T03:36:02.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.984327+0000", 2023-10-20T03:36:02.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129066 2023-10-20T03:36:02.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:02.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.984347+0000", 2023-10-20T03:36:02.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0186999999999998e-05 2023-10-20T03:36:02.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:02.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.002400+0000", 2023-10-20T03:36:02.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018053205999999999 2023-10-20T03:36:02.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.002510+0000", 2023-10-20T03:36:02.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010981599999999999 2023-10-20T03:36:02.464 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.464 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.465 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.465 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.3s2 71/56 ECSubWrite(tid=64, reqid=client.4570.0:576, at_version=71'47, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:02.465 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.006326+0000", 2023-10-20T03:36:02.465 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 9.2799048099999997, 2023-10-20T03:36:02.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030838075, 2023-10-20T03:36:02.466 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.466 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:02.466 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.467 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.467 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006326+0000", 2023-10-20T03:36:02.467 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.467 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.467 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.468 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006326+0000", 2023-10-20T03:36:02.468 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.468 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.468 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006328+0000", 2023-10-20T03:36:02.469 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2500000000000001e-06 2023-10-20T03:36:02.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006578+0000", 2023-10-20T03:36:02.470 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024969899999999998 2023-10-20T03:36:02.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.470 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.470 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006610+0000", 2023-10-20T03:36:02.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2499000000000002e-05 2023-10-20T03:36:02.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.471 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.471 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006627+0000", 2023-10-20T03:36:02.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6487e-05 2023-10-20T03:36:02.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006800+0000", 2023-10-20T03:36:02.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173439 2023-10-20T03:36:02.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.473 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:02.473 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006834+0000", 2023-10-20T03:36:02.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469000000000001e-05 2023-10-20T03:36:02.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.474 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:02.474 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.037070+0000", 2023-10-20T03:36:02.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030235952999999999 2023-10-20T03:36:02.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.475 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.475 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.037164+0000", 2023-10-20T03:36:02.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4278999999999996e-05 2023-10-20T03:36:02.475 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.476 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.476 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.476 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:611 3.ds0 3:b8ff490e:::smithi17039300-34:head [stat,delete] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-10-20T03:36:02.477 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:55.989324+0000", 2023-10-20T03:36:02.477 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.2969067899999995, 2023-10-20T03:36:02.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021838554, 2023-10-20T03:36:02.477 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.477 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:02.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:02.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:02.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:02.478 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 611 2023-10-20T03:36:02.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.479 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.479 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.479 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989324+0000", 2023-10-20T03:36:02.480 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.480 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.480 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.480 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989324+0000", 2023-10-20T03:36:02.481 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.481 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.481 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989327+0000", 2023-10-20T03:36:02.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.96e-06 2023-10-20T03:36:02.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.482 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989339+0000", 2023-10-20T03:36:02.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.261e-05 2023-10-20T03:36:02.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.483 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.483 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989376+0000", 2023-10-20T03:36:02.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6976e-05 2023-10-20T03:36:02.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.484 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.484 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989406+0000", 2023-10-20T03:36:02.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9873e-05 2023-10-20T03:36:02.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989561+0000", 2023-10-20T03:36:02.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015517300000000001 2023-10-20T03:36:02.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op must wait for map", 2023-10-20T03:36:02.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989579+0000", 2023-10-20T03:36:02.486 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7846000000000002e-05 2023-10-20T03:36:02.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.000958+0000", 2023-10-20T03:36:02.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01137907 2023-10-20T03:36:02.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.488 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:02.488 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.001262+0000", 2023-10-20T03:36:02.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030417399999999999 2023-10-20T03:36:02.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.489 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:02.489 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.002906+0000", 2023-10-20T03:36:02.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001643191 2023-10-20T03:36:02.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.490 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:02.490 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.007450+0000", 2023-10-20T03:36:02.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045443389999999997 2023-10-20T03:36:02.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:36:02.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.011024+0000", 2023-10-20T03:36:02.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035745400000000002 2023-10-20T03:36:02.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.492 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.492 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.011162+0000", 2023-10-20T03:36:02.492 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013780199999999999 2023-10-20T03:36:02.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.492 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.493 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:619 3.ds0 3:bbaca854:::smithi17039300-44:head [read 0~737280,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2023-10-20T03:36:02.493 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:57.009377+0000", 2023-10-20T03:36:02.493 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.2768531159999998, 2023-10-20T03:36:02.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019442797000000001, 2023-10-20T03:36:02.494 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.494 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:02.494 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:02.494 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:02.495 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:02.495 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 619 2023-10-20T03:36:02.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.495 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.496 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.496 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009377+0000", 2023-10-20T03:36:02.496 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009377+0000", 2023-10-20T03:36:02.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.498 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.498 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009381+0000", 2023-10-20T03:36:02.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2739999999999999e-06 2023-10-20T03:36:02.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.499 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009397+0000", 2023-10-20T03:36:02.499 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6549999999999999e-05 2023-10-20T03:36:02.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.500 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.500 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009442+0000", 2023-10-20T03:36:02.500 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4682999999999999e-05 2023-10-20T03:36:02.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.501 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009473+0000", 2023-10-20T03:36:02.501 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1163e-05 2023-10-20T03:36:02.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009642+0000", 2023-10-20T03:36:02.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016918200000000001 2023-10-20T03:36:02.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op must wait for map", 2023-10-20T03:36:02.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009668+0000", 2023-10-20T03:36:02.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6230000000000001e-05 2023-10-20T03:36:02.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.015833+0000", 2023-10-20T03:36:02.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0061643990000000001 2023-10-20T03:36:02.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:02.505 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.016114+0000", 2023-10-20T03:36:02.505 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028099199999999998 2023-10-20T03:36:02.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.505 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.506 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.028820+0000", 2023-10-20T03:36:02.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012706324 2023-10-20T03:36:02.506 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.506 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.506 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.507 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:620 3.ds0 3:bbaca854:::smithi17039300-44:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2023-10-20T03:36:02.507 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:57.009528+0000", 2023-10-20T03:36:02.507 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.2767024200000003, 2023-10-20T03:36:02.507 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021296131999999999, 2023-10-20T03:36:02.508 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.508 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:02.508 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:02.508 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:02.508 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:02.509 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 620 2023-10-20T03:36:02.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.509 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.509 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009528+0000", 2023-10-20T03:36:02.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009528+0000", 2023-10-20T03:36:02.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009530+0000", 2023-10-20T03:36:02.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.637e-06 2023-10-20T03:36:02.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009541+0000", 2023-10-20T03:36:02.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.133e-05 2023-10-20T03:36:02.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.513 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.513 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009564+0000", 2023-10-20T03:36:02.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3220000000000001e-05 2023-10-20T03:36:02.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009576+0000", 2023-10-20T03:36:02.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1419e-05 2023-10-20T03:36:02.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009879+0000", 2023-10-20T03:36:02.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030328100000000003 2023-10-20T03:36:02.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:02.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009902+0000", 2023-10-20T03:36:02.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3507e-05 2023-10-20T03:36:02.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.517 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.024765+0000", 2023-10-20T03:36:02.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014862548 2023-10-20T03:36:02.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:02.518 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.025046+0000", 2023-10-20T03:36:02.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028078100000000002 2023-10-20T03:36:02.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.030824+0000", 2023-10-20T03:36:02.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005778409 2023-10-20T03:36:02.519 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.519 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.520 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.520 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:621 3.ds0 3:bbaca854:::smithi17039300-44:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2023-10-20T03:36:02.520 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:57.009600+0000", 2023-10-20T03:36:02.521 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.2766307110000001, 2023-10-20T03:36:02.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.02940736, 2023-10-20T03:36:02.521 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:02.521 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:02.521 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:02.521 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:02.522 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:02.522 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 621 2023-10-20T03:36:02.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.522 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:02.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:02.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009600+0000", 2023-10-20T03:36:02.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:02.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009600+0000", 2023-10-20T03:36:02.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:02.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:02.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009601+0000", 2023-10-20T03:36:02.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7700000000000003e-07 2023-10-20T03:36:02.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:02.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009606+0000", 2023-10-20T03:36:02.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6699999999999999e-06 2023-10-20T03:36:02.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:02.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009617+0000", 2023-10-20T03:36:02.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0643e-05 2023-10-20T03:36:02.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:02.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009625+0000", 2023-10-20T03:36:02.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2130000000000008e-06 2023-10-20T03:36:02.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.010959+0000", 2023-10-20T03:36:02.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001334277 2023-10-20T03:36:02.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:02.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.010984+0000", 2023-10-20T03:36:02.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4487000000000001e-05 2023-10-20T03:36:02.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:02.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.030984+0000", 2023-10-20T03:36:02.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019999751999999999 2023-10-20T03:36:02.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:02.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.031305+0000", 2023-10-20T03:36:02.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032115399999999998 2023-10-20T03:36:02.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:02.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:02.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:02.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.039007+0000", 2023-10-20T03:36:02.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0077022870000000004 2023-10-20T03:36:02.532 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.532 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.533 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.533 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:02.533 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:02.533 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:02.535 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-10-20T03:36:02.535 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:36:02.532+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:36:02.636 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:02.633+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:02.712 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:02.712 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:36:02.713 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:36:02.713 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:02.727 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-10-20T03:36:02.736 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:02.733+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:02.771 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2023-10-20T03:36:02.777 INFO:tasks.rados.rados.0.smithi170.stdout:270: done (0 left) 2023-10-20T03:36:02.778 INFO:tasks.rados.rados.0.smithi170.stdout:271: delete oid 1 current snap is 23 2023-10-20T03:36:02.785 INFO:tasks.rados.rados.0.smithi170.stdout:271: done (0 left) 2023-10-20T03:36:02.785 INFO:tasks.rados.rados.0.smithi170.stdout:272: copy_from oid 27 from oid 47 current snap is 23 2023-10-20T03:36:02.785 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 10 current snap is 23 2023-10-20T03:36:02.786 INFO:tasks.rados.rados.0.smithi170.stdout:273: seq_num 119 ranges {1163264=458752,1622016=507904,2129920=507904,2637824=458752,3096576=81920} 2023-10-20T03:36:02.811 INFO:tasks.rados.rados.0.smithi170.stdout:273: writing smithi17039300-10 from 1163264 to 1622016 tid 1 2023-10-20T03:36:02.818 INFO:tasks.rados.rados.0.smithi170.stdout:273: writing smithi17039300-10 from 1622016 to 2129920 tid 2 2023-10-20T03:36:02.825 INFO:tasks.rados.rados.0.smithi170.stdout:273: writing smithi17039300-10 from 2129920 to 2637824 tid 3 2023-10-20T03:36:02.832 INFO:tasks.rados.rados.0.smithi170.stdout:273: writing smithi17039300-10 from 2637824 to 3096576 tid 4 2023-10-20T03:36:02.834 INFO:tasks.rados.rados.0.smithi170.stdout:273: writing smithi17039300-10 from 3096576 to 3178496 tid 5 2023-10-20T03:36:02.834 INFO:tasks.rados.rados.0.smithi170.stdout:274: rmattr oid 17 current snap is 23 2023-10-20T03:36:02.834 INFO:tasks.rados.rados.0.smithi170.stdout:272: finishing copy_from to smithi17039300-27 2023-10-20T03:36:02.834 INFO:tasks.rados.rados.0.smithi170.stdout:272: got expected ENOENT (src dne) 2023-10-20T03:36:02.834 INFO:tasks.rados.rados.0.smithi170.stdout:272: finishing copy_from racing read to smithi17039300-27 2023-10-20T03:36:02.835 INFO:tasks.rados.rados.0.smithi170.stdout:272: done (2 left) 2023-10-20T03:36:02.835 INFO:tasks.rados.rados.0.smithi170.stdout:275: snap_create 2023-10-20T03:36:02.836 INFO:tasks.rados.rados.0.smithi170.stdout:273: finishing write tid 1 to smithi17039300-10 2023-10-20T03:36:02.837 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:02.835+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:02.838 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 17 v 58 (ObjNum 114 snap 19 seq_num 114) dirty exists 2023-10-20T03:36:02.846 INFO:tasks.rados.rados.0.smithi170.stdout:273: finishing write tid 2 to smithi17039300-10 2023-10-20T03:36:02.846 INFO:tasks.rados.rados.0.smithi170.stdout:273: finishing write tid 3 to smithi17039300-10 2023-10-20T03:36:02.848 INFO:tasks.rados.rados.0.smithi170.stdout:273: finishing write tid 4 to smithi17039300-10 2023-10-20T03:36:02.849 INFO:tasks.rados.rados.0.smithi170.stdout:273: finishing write tid 5 to smithi17039300-10 2023-10-20T03:36:02.849 INFO:tasks.rados.rados.0.smithi170.stdout:273: finishing write tid 6 to smithi17039300-10 2023-10-20T03:36:02.852 INFO:tasks.rados.rados.0.smithi170.stdout:273: finishing write tid 7 to smithi17039300-10 2023-10-20T03:36:02.852 INFO:tasks.rados.rados.0.smithi170.stdout:273: oid 10 updating version 0 to 64 2023-10-20T03:36:02.852 INFO:tasks.rados.rados.0.smithi170.stdout:273: oid 10 version 64 is already newer than 63 2023-10-20T03:36:02.853 INFO:tasks.rados.rados.0.smithi170.stdout:273: oid 10 version 64 is already newer than 61 2023-10-20T03:36:02.853 INFO:tasks.rados.rados.0.smithi170.stdout:273: oid 10 updating version 64 to 65 2023-10-20T03:36:02.853 INFO:tasks.rados.rados.0.smithi170.stdout:273: oid 10 version 65 is already newer than 60 2023-10-20T03:36:02.853 INFO:tasks.rados.rados.0.smithi170.stdout:273: oid 10 version 65 is already newer than 62 2023-10-20T03:36:02.853 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 10 v 65 (ObjNum 119 snap 23 seq_num 119) dirty exists 2023-10-20T03:36:02.854 INFO:tasks.rados.rados.0.smithi170.stdout:273: left oid 10 (ObjNum 119 snap 23 seq_num 119) 2023-10-20T03:36:02.863 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:02.863 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:36:02.863 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:36:02.863 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:36:02.864 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:02.878 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-10-20T03:36:02.938 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:02.936+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:03.004 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:03.005 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:36:03.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:36:03.005 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2023-10-20T03:36:03.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.005 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=23, reqid=client.4570.0:208, at_version=37'14, trim_to=0'0, roll_forward_to=36'12))", 2023-10-20T03:36:03.005 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:19.272177+0000", 2023-10-20T03:36:03.006 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 43.728906444000003, 2023-10-20T03:36:03.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010004097, 2023-10-20T03:36:03.006 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.006 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.007 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.007 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.007 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272177+0000", 2023-10-20T03:36:03.007 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.008 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.008 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272177+0000", 2023-10-20T03:36:03.008 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.009 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.009 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272177+0000", 2023-10-20T03:36:03.009 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5800000000000001e-07 2023-10-20T03:36:03.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.010 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.010 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272691+0000", 2023-10-20T03:36:03.010 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051351599999999999 2023-10-20T03:36:03.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.011 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.011 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272709+0000", 2023-10-20T03:36:03.011 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8108999999999999e-05 2023-10-20T03:36:03.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.012 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.012 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272738+0000", 2023-10-20T03:36:03.012 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9246000000000001e-05 2023-10-20T03:36:03.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.013 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.013 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272855+0000", 2023-10-20T03:36:03.013 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011678 2023-10-20T03:36:03.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.014 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.014 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.272869+0000", 2023-10-20T03:36:03.014 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4409999999999999e-05 2023-10-20T03:36:03.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.015 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.015 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282065+0000", 2023-10-20T03:36:03.015 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0091960219999999999 2023-10-20T03:36:03.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.016 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.016 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:19.282181+0000", 2023-10-20T03:36:03.016 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011575599999999999 2023-10-20T03:36:03.016 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.017 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.017 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.017 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 43/31 ECSubWrite(tid=18, reqid=client.4570.0:316, at_version=43'8, trim_to=0'0, roll_forward_to=34'6))", 2023-10-20T03:36:03.018 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.368510+0000", 2023-10-20T03:36:03.018 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 37.632573585000003, 2023-10-20T03:36:03.018 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010086807, 2023-10-20T03:36:03.018 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.018 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.019 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.019 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.019 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368510+0000", 2023-10-20T03:36:03.019 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.020 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.020 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.020 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368510+0000", 2023-10-20T03:36:03.020 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.021 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.021 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368510+0000", 2023-10-20T03:36:03.021 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7999999999999995e-07 2023-10-20T03:36:03.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.022 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.022 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368574+0000", 2023-10-20T03:36:03.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3555999999999999e-05 2023-10-20T03:36:03.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.023 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.023 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368612+0000", 2023-10-20T03:36:03.023 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8386999999999997e-05 2023-10-20T03:36:03.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.024 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.024 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368638+0000", 2023-10-20T03:36:03.024 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5775999999999999e-05 2023-10-20T03:36:03.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.024 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.025 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.025 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368887+0000", 2023-10-20T03:36:03.025 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024854500000000003 2023-10-20T03:36:03.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.025 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.025 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.026 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.368910+0000", 2023-10-20T03:36:03.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.353e-05 2023-10-20T03:36:03.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.026 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.026 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.378493+0000", 2023-10-20T03:36:03.027 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0095828070000000005 2023-10-20T03:36:03.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.378597+0000", 2023-10-20T03:36:03.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000103626 2023-10-20T03:36:03.028 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.028 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.029 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.029 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 43/31 ECSubWrite(tid=19, reqid=client.4570.0:317, at_version=43'9, trim_to=0'0, roll_forward_to=34'6))", 2023-10-20T03:36:03.029 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.369969+0000", 2023-10-20T03:36:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 37.631114812, 2023-10-20T03:36:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010764182000000001, 2023-10-20T03:36:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369969+0000", 2023-10-20T03:36:03.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369969+0000", 2023-10-20T03:36:03.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.369971+0000", 2023-10-20T03:36:03.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8360000000000001e-06 2023-10-20T03:36:03.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.034 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370018+0000", 2023-10-20T03:36:03.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7976999999999998e-05 2023-10-20T03:36:03.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.035 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370050+0000", 2023-10-20T03:36:03.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1668999999999998e-05 2023-10-20T03:36:03.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.036 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370078+0000", 2023-10-20T03:36:03.036 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8019999999999999e-05 2023-10-20T03:36:03.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.037 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.371731+0000", 2023-10-20T03:36:03.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0016532039999999999 2023-10-20T03:36:03.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.371760+0000", 2023-10-20T03:36:03.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.904e-05 2023-10-20T03:36:03.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.380637+0000", 2023-10-20T03:36:03.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0088761570000000008 2023-10-20T03:36:03.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.040 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.380733+0000", 2023-10-20T03:36:03.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6279000000000004e-05 2023-10-20T03:36:03.040 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.040 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.040 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.041 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.041 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 43/31 ECSubWrite(tid=20, reqid=client.4570.0:318, at_version=43'10, trim_to=0'0, roll_forward_to=34'6))", 2023-10-20T03:36:03.041 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:25.370315+0000", 2023-10-20T03:36:03.041 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 37.630768854999999, 2023-10-20T03:36:03.041 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010484894, 2023-10-20T03:36:03.042 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.042 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.042 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.042 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.042 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.043 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370315+0000", 2023-10-20T03:36:03.043 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.043 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.043 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.044 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370315+0000", 2023-10-20T03:36:03.044 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.044 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.044 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.045 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370316+0000", 2023-10-20T03:36:03.045 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4699999999999999e-06 2023-10-20T03:36:03.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.045 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.045 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.046 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370356+0000", 2023-10-20T03:36:03.046 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9996000000000001e-05 2023-10-20T03:36:03.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.046 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370382+0000", 2023-10-20T03:36:03.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6310000000000001e-05 2023-10-20T03:36:03.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.047 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.370403+0000", 2023-10-20T03:36:03.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0784e-05 2023-10-20T03:36:03.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.374885+0000", 2023-10-20T03:36:03.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0044819090000000001 2023-10-20T03:36:03.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.374981+0000", 2023-10-20T03:36:03.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5618999999999994e-05 2023-10-20T03:36:03.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.050 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.050 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.380741+0000", 2023-10-20T03:36:03.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057606469999999998 2023-10-20T03:36:03.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:25.380800+0000", 2023-10-20T03:36:03.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8158999999999997e-05 2023-10-20T03:36:03.052 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.052 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.052 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.053 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:03.054 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.387934+0000", 2023-10-20T03:36:03.054 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 36.613149872999998, 2023-10-20T03:36:03.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033593076999999999, 2023-10-20T03:36:03.054 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.054 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.055 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387934+0000", 2023-10-20T03:36:03.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387934+0000", 2023-10-20T03:36:03.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387934+0000", 2023-10-20T03:36:03.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7e-07 2023-10-20T03:36:03.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.058 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388584+0000", 2023-10-20T03:36:03.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00064982500000000003 2023-10-20T03:36:03.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388724+0000", 2023-10-20T03:36:03.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013997200000000001 2023-10-20T03:36:03.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388764+0000", 2023-10-20T03:36:03.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0352999999999998e-05 2023-10-20T03:36:03.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388939+0000", 2023-10-20T03:36:03.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017499200000000001 2023-10-20T03:36:03.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388956+0000", 2023-10-20T03:36:03.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6990000000000002e-05 2023-10-20T03:36:03.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421453+0000", 2023-10-20T03:36:03.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032496615999999999 2023-10-20T03:36:03.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421527+0000", 2023-10-20T03:36:03.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3959000000000001e-05 2023-10-20T03:36:03.066 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.066 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.067 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.067 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/31 ECSubWrite(tid=22, reqid=client.4570.0:340, at_version=44'12, trim_to=0'0, roll_forward_to=43'10))", 2023-10-20T03:36:03.067 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:36:03.068 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 36.602174052999999, 2023-10-20T03:36:03.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022077208000000001, 2023-10-20T03:36:03.068 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.068 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.068 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:36:03.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:36:03.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398910+0000", 2023-10-20T03:36:03.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6400000000000002e-07 2023-10-20T03:36:03.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399138+0000", 2023-10-20T03:36:03.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022810800000000001 2023-10-20T03:36:03.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399169+0000", 2023-10-20T03:36:03.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0593000000000003e-05 2023-10-20T03:36:03.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399190+0000", 2023-10-20T03:36:03.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.12e-05 2023-10-20T03:36:03.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399353+0000", 2023-10-20T03:36:03.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016285200000000001 2023-10-20T03:36:03.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399378+0000", 2023-10-20T03:36:03.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5077000000000001e-05 2023-10-20T03:36:03.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420889+0000", 2023-10-20T03:36:03.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021511188000000001 2023-10-20T03:36:03.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420987+0000", 2023-10-20T03:36:03.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7626000000000003e-05 2023-10-20T03:36:03.078 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.078 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.079 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.079 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:03.079 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.401022+0000", 2023-10-20T03:36:03.080 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 36.600061019000002, 2023-10-20T03:36:03.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020571461999999999, 2023-10-20T03:36:03.080 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.080 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.080 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401022+0000", 2023-10-20T03:36:03.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401022+0000", 2023-10-20T03:36:03.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401024+0000", 2023-10-20T03:36:03.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.877e-06 2023-10-20T03:36:03.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401473+0000", 2023-10-20T03:36:03.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044877699999999998 2023-10-20T03:36:03.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401507+0000", 2023-10-20T03:36:03.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4209999999999999e-05 2023-10-20T03:36:03.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401543+0000", 2023-10-20T03:36:03.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6003000000000001e-05 2023-10-20T03:36:03.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401767+0000", 2023-10-20T03:36:03.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022351200000000001 2023-10-20T03:36:03.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401793+0000", 2023-10-20T03:36:03.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6529e-05 2023-10-20T03:36:03.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421560+0000", 2023-10-20T03:36:03.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019766305000000001 2023-10-20T03:36:03.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421594+0000", 2023-10-20T03:36:03.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4248999999999997e-05 2023-10-20T03:36:03.090 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.090 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.091 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.091 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 44/31 ECSubWrite(tid=53, reqid=client.4570.0:335, at_version=44'20, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:03.091 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.402960+0000", 2023-10-20T03:36:03.092 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 36.598122986, 2023-10-20T03:36:03.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019889799999999999, 2023-10-20T03:36:03.092 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.092 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.092 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402960+0000", 2023-10-20T03:36:03.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402960+0000", 2023-10-20T03:36:03.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402962+0000", 2023-10-20T03:36:03.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8470000000000001e-06 2023-10-20T03:36:03.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403065+0000", 2023-10-20T03:36:03.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000102179 2023-10-20T03:36:03.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403099+0000", 2023-10-20T03:36:03.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4813e-05 2023-10-20T03:36:03.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403122+0000", 2023-10-20T03:36:03.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2226e-05 2023-10-20T03:36:03.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419074+0000", 2023-10-20T03:36:03.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015952643999999998 2023-10-20T03:36:03.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419093+0000", 2023-10-20T03:36:03.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9089e-05 2023-10-20T03:36:03.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.422750+0000", 2023-10-20T03:36:03.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0036562959999999999 2023-10-20T03:36:03.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.422850+0000", 2023-10-20T03:36:03.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010070599999999999 2023-10-20T03:36:03.102 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.102 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.102 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.103 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 44/31 ECSubWrite(tid=54, reqid=client.4570.0:336, at_version=44'21, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:03.103 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.404639+0000", 2023-10-20T03:36:03.103 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 36.596444386000002, 2023-10-20T03:36:03.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018626967000000001, 2023-10-20T03:36:03.103 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.104 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.104 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404639+0000", 2023-10-20T03:36:03.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404639+0000", 2023-10-20T03:36:03.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404641+0000", 2023-10-20T03:36:03.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7969999999999999e-06 2023-10-20T03:36:03.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404707+0000", 2023-10-20T03:36:03.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6523e-05 2023-10-20T03:36:03.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404741+0000", 2023-10-20T03:36:03.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4078999999999997e-05 2023-10-20T03:36:03.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404764+0000", 2023-10-20T03:36:03.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.209e-05 2023-10-20T03:36:03.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420947+0000", 2023-10-20T03:36:03.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016183831999999999 2023-10-20T03:36:03.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420971+0000", 2023-10-20T03:36:03.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.332e-05 2023-10-20T03:36:03.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.423172+0000", 2023-10-20T03:36:03.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002200867 2023-10-20T03:36:03.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.423266+0000", 2023-10-20T03:36:03.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4458999999999998e-05 2023-10-20T03:36:03.114 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.114 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.114 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.115 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:36:03.115 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:30.392373+0000", 2023-10-20T03:36:03.115 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 32.608710174999999, 2023-10-20T03:36:03.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038515463, 2023-10-20T03:36:03.116 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.116 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.116 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.392373+0000", 2023-10-20T03:36:03.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.392373+0000", 2023-10-20T03:36:03.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.392374+0000", 2023-10-20T03:36:03.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3e-07 2023-10-20T03:36:03.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394109+0000", 2023-10-20T03:36:03.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001734977 2023-10-20T03:36:03.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394151+0000", 2023-10-20T03:36:03.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1872999999999997e-05 2023-10-20T03:36:03.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394177+0000", 2023-10-20T03:36:03.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5636e-05 2023-10-20T03:36:03.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394320+0000", 2023-10-20T03:36:03.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014345499999999999 2023-10-20T03:36:03.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394348+0000", 2023-10-20T03:36:03.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7523999999999999e-05 2023-10-20T03:36:03.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.430793+0000", 2023-10-20T03:36:03.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.036445549000000001 2023-10-20T03:36:03.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.430889+0000", 2023-10-20T03:36:03.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5718999999999996e-05 2023-10-20T03:36:03.126 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.126 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.126 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.127 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 49/31 ECSubWrite(tid=24, reqid=client.4570.0:392, at_version=49'14, trim_to=0'0, roll_forward_to=44'12))", 2023-10-20T03:36:03.127 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.388889+0000", 2023-10-20T03:36:03.127 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 31.612194814999999, 2023-10-20T03:36:03.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0086355749999999995, 2023-10-20T03:36:03.128 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.128 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.128 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388889+0000", 2023-10-20T03:36:03.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388889+0000", 2023-10-20T03:36:03.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.130 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388889+0000", 2023-10-20T03:36:03.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.13e-07 2023-10-20T03:36:03.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.388970+0000", 2023-10-20T03:36:03.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1123000000000002e-05 2023-10-20T03:36:03.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389004+0000", 2023-10-20T03:36:03.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3946000000000003e-05 2023-10-20T03:36:03.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389029+0000", 2023-10-20T03:36:03.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4187000000000001e-05 2023-10-20T03:36:03.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.134 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389281+0000", 2023-10-20T03:36:03.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025258799999999999 2023-10-20T03:36:03.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.135 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.135 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.389311+0000", 2023-10-20T03:36:03.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.993e-05 2023-10-20T03:36:03.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.397420+0000", 2023-10-20T03:36:03.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0081085359999999995 2023-10-20T03:36:03.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.397524+0000", 2023-10-20T03:36:03.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104652 2023-10-20T03:36:03.138 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.138 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.138 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.139 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 53/31 ECSubWrite(tid=33, reqid=osd.8.0:33, at_version=53'24, trim_to=0'0, roll_forward_to=52'22))", 2023-10-20T03:36:03.139 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:35.431073+0000", 2023-10-20T03:36:03.139 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 27.570010407000002, 2023-10-20T03:36:03.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0076161019999999996, 2023-10-20T03:36:03.139 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.140 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.140 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431073+0000", 2023-10-20T03:36:03.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431073+0000", 2023-10-20T03:36:03.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431074+0000", 2023-10-20T03:36:03.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4999999999999998e-07 2023-10-20T03:36:03.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431152+0000", 2023-10-20T03:36:03.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8329000000000002e-05 2023-10-20T03:36:03.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431270+0000", 2023-10-20T03:36:03.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000118403 2023-10-20T03:36:03.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431298+0000", 2023-10-20T03:36:03.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7512999999999999e-05 2023-10-20T03:36:03.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.147 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431585+0000", 2023-10-20T03:36:03.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028691399999999998 2023-10-20T03:36:03.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.147 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.431614+0000", 2023-10-20T03:36:03.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9280000000000001e-05 2023-10-20T03:36:03.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.438579+0000", 2023-10-20T03:36:03.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0069652580000000002 2023-10-20T03:36:03.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.438689+0000", 2023-10-20T03:36:03.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109955 2023-10-20T03:36:03.150 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.150 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.150 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.151 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 60/56 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=60'45, trim_to=0'0, roll_forward_to=57'43))", 2023-10-20T03:36:03.151 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.830961+0000", 2023-10-20T03:36:03.151 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.170122422999999, 2023-10-20T03:36:03.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090787750000000007, 2023-10-20T03:36:03.151 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.151 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.152 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.830961+0000", 2023-10-20T03:36:03.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.830961+0000", 2023-10-20T03:36:03.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.154 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.830962+0000", 2023-10-20T03:36:03.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.63e-07 2023-10-20T03:36:03.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.831005+0000", 2023-10-20T03:36:03.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3516000000000001e-05 2023-10-20T03:36:03.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.831037+0000", 2023-10-20T03:36:03.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2153999999999997e-05 2023-10-20T03:36:03.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.831066+0000", 2023-10-20T03:36:03.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9043e-05 2023-10-20T03:36:03.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.831253+0000", 2023-10-20T03:36:03.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000186711 2023-10-20T03:36:03.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.831286+0000", 2023-10-20T03:36:03.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3396999999999999e-05 2023-10-20T03:36:03.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.839932+0000", 2023-10-20T03:36:03.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0086460649999999997 2023-10-20T03:36:03.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.840040+0000", 2023-10-20T03:36:03.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107426 2023-10-20T03:36:03.161 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.161 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.162 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.162 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 60/56 ECSubWrite(tid=28, reqid=osd.1.0:28, at_version=60'47, trim_to=0'0, roll_forward_to=57'43))", 2023-10-20T03:36:03.163 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.834067+0000", 2023-10-20T03:36:03.163 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.1670169, 2023-10-20T03:36:03.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010530794, 2023-10-20T03:36:03.163 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.163 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.164 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.834067+0000", 2023-10-20T03:36:03.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.834067+0000", 2023-10-20T03:36:03.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.834069+0000", 2023-10-20T03:36:03.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0770000000000002e-06 2023-10-20T03:36:03.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.834123+0000", 2023-10-20T03:36:03.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4611999999999998e-05 2023-10-20T03:36:03.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.834166+0000", 2023-10-20T03:36:03.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2370000000000003e-05 2023-10-20T03:36:03.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.834190+0000", 2023-10-20T03:36:03.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4573e-05 2023-10-20T03:36:03.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.170 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.836006+0000", 2023-10-20T03:36:03.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00181566 2023-10-20T03:36:03.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.836033+0000", 2023-10-20T03:36:03.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6973000000000001e-05 2023-10-20T03:36:03.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.844509+0000", 2023-10-20T03:36:03.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084763700000000004 2023-10-20T03:36:03.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.844597+0000", 2023-10-20T03:36:03.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8158999999999994e-05 2023-10-20T03:36:03.173 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.173 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.173 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.174 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 60/31 ECSubWrite(tid=39, reqid=osd.8.0:39, at_version=60'30, trim_to=0'0, roll_forward_to=56'28))", 2023-10-20T03:36:03.174 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.843614+0000", 2023-10-20T03:36:03.174 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.157469538000001, 2023-10-20T03:36:03.175 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0074417670000000002, 2023-10-20T03:36:03.175 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.175 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.175 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.176 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.176 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843614+0000", 2023-10-20T03:36:03.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.177 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.177 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.177 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843614+0000", 2023-10-20T03:36:03.177 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.178 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.178 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.178 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843615+0000", 2023-10-20T03:36:03.178 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7999999999999995e-07 2023-10-20T03:36:03.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.179 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.179 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843670+0000", 2023-10-20T03:36:03.179 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5442999999999998e-05 2023-10-20T03:36:03.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843701+0000", 2023-10-20T03:36:03.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0985999999999998e-05 2023-10-20T03:36:03.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.181 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.843727+0000", 2023-10-20T03:36:03.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5497e-05 2023-10-20T03:36:03.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.181 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.181 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.844678+0000", 2023-10-20T03:36:03.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00095127600000000003 2023-10-20T03:36:03.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.844706+0000", 2023-10-20T03:36:03.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8002999999999999e-05 2023-10-20T03:36:03.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.850934+0000", 2023-10-20T03:36:03.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062281630000000001 2023-10-20T03:36:03.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.184 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.851056+0000", 2023-10-20T03:36:03.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000121719 2023-10-20T03:36:03.184 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.185 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.185 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.185 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 60/56 ECSubWrite(tid=29, reqid=osd.1.0:28, at_version=0'0, trim_to=0'0, roll_forward_to=60'47))", 2023-10-20T03:36:03.185 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.847024+0000", 2023-10-20T03:36:03.186 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.154059756999999, 2023-10-20T03:36:03.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0066434390000000001, 2023-10-20T03:36:03.186 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.186 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.186 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847024+0000", 2023-10-20T03:36:03.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847024+0000", 2023-10-20T03:36:03.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847026+0000", 2023-10-20T03:36:03.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0099999999999998e-06 2023-10-20T03:36:03.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.190 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.190 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847066+0000", 2023-10-20T03:36:03.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0111999999999998e-05 2023-10-20T03:36:03.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.191 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847100+0000", 2023-10-20T03:36:03.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4397000000000003e-05 2023-10-20T03:36:03.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847148+0000", 2023-10-20T03:36:03.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.816e-05 2023-10-20T03:36:03.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847316+0000", 2023-10-20T03:36:03.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016719799999999999 2023-10-20T03:36:03.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.847345+0000", 2023-10-20T03:36:03.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.898e-05 2023-10-20T03:36:03.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.194 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.853568+0000", 2023-10-20T03:36:03.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.006223096 2023-10-20T03:36:03.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.195 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.195 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.853667+0000", 2023-10-20T03:36:03.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9486000000000003e-05 2023-10-20T03:36:03.196 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.196 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.196 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.197 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 71/31 ECSubWrite(tid=52, reqid=client.4570.0:589, at_version=71'24, trim_to=0'0, roll_forward_to=61'23))", 2023-10-20T03:36:03.197 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.004084+0000", 2023-10-20T03:36:03.197 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 9.9969998709999999, 2023-10-20T03:36:03.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041882438000000001, 2023-10-20T03:36:03.198 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.198 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.198 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.004084+0000", 2023-10-20T03:36:03.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.004084+0000", 2023-10-20T03:36:03.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.201 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.201 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.004086+0000", 2023-10-20T03:36:03.201 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8499999999999998e-06 2023-10-20T03:36:03.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.005054+0000", 2023-10-20T03:36:03.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00096745999999999996 2023-10-20T03:36:03.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.203 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.203 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.005115+0000", 2023-10-20T03:36:03.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0888999999999999e-05 2023-10-20T03:36:03.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.204 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.204 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.005144+0000", 2023-10-20T03:36:03.204 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9122999999999999e-05 2023-10-20T03:36:03.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.005300+0000", 2023-10-20T03:36:03.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000156269 2023-10-20T03:36:03.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.206 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.206 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.005316+0000", 2023-10-20T03:36:03.206 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6277e-05 2023-10-20T03:36:03.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.207 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.207 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.045929+0000", 2023-10-20T03:36:03.207 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040612662000000001 2023-10-20T03:36:03.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.207 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.208 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.208 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.045966+0000", 2023-10-20T03:36:03.208 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6908e-05 2023-10-20T03:36:03.208 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.208 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.209 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.209 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 71/31 ECSubWrite(tid=53, reqid=client.4570.0:590, at_version=71'25, trim_to=0'0, roll_forward_to=61'23))", 2023-10-20T03:36:03.209 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.013561+0000", 2023-10-20T03:36:03.209 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 9.9875229319999992, 2023-10-20T03:36:03.210 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032941350000000001, 2023-10-20T03:36:03.210 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.210 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.210 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.211 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.211 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.013561+0000", 2023-10-20T03:36:03.211 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.212 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.212 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.013561+0000", 2023-10-20T03:36:03.212 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.212 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.213 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.013563+0000", 2023-10-20T03:36:03.213 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5600000000000001e-06 2023-10-20T03:36:03.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.213 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.214 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.014144+0000", 2023-10-20T03:36:03.214 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00058107500000000002 2023-10-20T03:36:03.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.014197+0000", 2023-10-20T03:36:03.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3257000000000002e-05 2023-10-20T03:36:03.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.216 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.014223+0000", 2023-10-20T03:36:03.216 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6006000000000001e-05 2023-10-20T03:36:03.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.216 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.217 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.024055+0000", 2023-10-20T03:36:03.217 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0098312320000000005 2023-10-20T03:36:03.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.217 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.218 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.024090+0000", 2023-10-20T03:36:03.218 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5309999999999999e-05 2023-10-20T03:36:03.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.219 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.219 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.046390+0000", 2023-10-20T03:36:03.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022299864999999999 2023-10-20T03:36:03.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.220 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.220 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.046502+0000", 2023-10-20T03:36:03.220 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112045 2023-10-20T03:36:03.220 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.220 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.221 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.221 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 71/31 ECSubWrite(tid=54, reqid=client.4570.0:591, at_version=71'26, trim_to=0'0, roll_forward_to=61'23))", 2023-10-20T03:36:03.221 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.035029+0000", 2023-10-20T03:36:03.222 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 9.9660549249999999, 2023-10-20T03:36:03.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012217623, 2023-10-20T03:36:03.222 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.222 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.222 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.035029+0000", 2023-10-20T03:36:03.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.035029+0000", 2023-10-20T03:36:03.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.035031+0000", 2023-10-20T03:36:03.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7140000000000002e-06 2023-10-20T03:36:03.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.035295+0000", 2023-10-20T03:36:03.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026349400000000002 2023-10-20T03:36:03.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.227 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.035343+0000", 2023-10-20T03:36:03.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7790000000000002e-05 2023-10-20T03:36:03.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.035370+0000", 2023-10-20T03:36:03.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.762e-05 2023-10-20T03:36:03.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.228 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.040863+0000", 2023-10-20T03:36:03.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0054923350000000001 2023-10-20T03:36:03.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.040887+0000", 2023-10-20T03:36:03.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4683000000000001e-05 2023-10-20T03:36:03.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.047207+0000", 2023-10-20T03:36:03.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0063196069999999997 2023-10-20T03:36:03.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.047246+0000", 2023-10-20T03:36:03.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9379999999999999e-05 2023-10-20T03:36:03.232 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.232 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.232 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.232 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 74/56 ECSubWrite(tid=54, reqid=client.4570.0:611, at_version=74'53, trim_to=0'0, roll_forward_to=61'51))", 2023-10-20T03:36:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:56.002720+0000", 2023-10-20T03:36:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.9983639200000001, 2023-10-20T03:36:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0078510539999999997, 2023-10-20T03:36:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.233 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.234 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.002720+0000", 2023-10-20T03:36:03.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.002720+0000", 2023-10-20T03:36:03.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.002720+0000", 2023-10-20T03:36:03.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7999999999999995e-07 2023-10-20T03:36:03.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.237 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.237 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.002811+0000", 2023-10-20T03:36:03.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0296000000000004e-05 2023-10-20T03:36:03.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.238 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.238 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.002848+0000", 2023-10-20T03:36:03.238 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7586000000000002e-05 2023-10-20T03:36:03.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.002875+0000", 2023-10-20T03:36:03.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7226e-05 2023-10-20T03:36:03.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.003114+0000", 2023-10-20T03:36:03.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00023878199999999999 2023-10-20T03:36:03.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.003148+0000", 2023-10-20T03:36:03.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4319999999999997e-05 2023-10-20T03:36:03.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.242 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.242 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.010499+0000", 2023-10-20T03:36:03.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0073508209999999996 2023-10-20T03:36:03.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.010571+0000", 2023-10-20T03:36:03.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1342999999999998e-05 2023-10-20T03:36:03.243 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.244 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.244 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.244 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.244 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.244 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:03.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-10-20T03:36:03.247 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:36:03.244+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:36:03.347 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:03.345+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:03.382 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:36:03.426 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:03.427 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:36:03.427 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:36:03.427 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:03.441 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-10-20T03:36:03.490 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:03.488+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:03.542 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:36:03.543 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:36:03.548 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:03.544+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:03.562 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:03.562 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:36:03.563 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:36:03.563 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:36:03.563 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:03.563 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:36:03.564 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:36:03.577 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-10-20T03:36:03.586 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:36:03.587 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:36:03.608 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:36:03.609 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:36:03.629 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:36:03.629 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:36:03.649 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:03.647+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:03.649 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:36:03.650 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:36:03.650 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:36:03.673 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:36:03.674 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:36:03.747 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:03.748 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:36:03.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:36:03.748 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2023-10-20T03:36:03.748 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.749 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.0s6 44/31 ECSubWrite(tid=18, reqid=client.4570.0:323, at_version=44'9, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:36:03.749 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.361041+0000", 2023-10-20T03:36:03.749 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 37.385036952999997, 2023-10-20T03:36:03.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015954300000000001, 2023-10-20T03:36:03.749 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.749 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.750 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.750 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361041+0000", 2023-10-20T03:36:03.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.751 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.751 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361041+0000", 2023-10-20T03:36:03.751 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.752 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.752 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361041+0000", 2023-10-20T03:36:03.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2699999999999998e-07 2023-10-20T03:36:03.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.753 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361502+0000", 2023-10-20T03:36:03.753 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046024999999999999 2023-10-20T03:36:03.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.754 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361532+0000", 2023-10-20T03:36:03.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0286000000000001e-05 2023-10-20T03:36:03.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.755 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.755 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361548+0000", 2023-10-20T03:36:03.755 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5797000000000001e-05 2023-10-20T03:36:03.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.756 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.756 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361674+0000", 2023-10-20T03:36:03.756 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012640500000000001 2023-10-20T03:36:03.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.756 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.757 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.757 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.361693+0000", 2023-10-20T03:36:03.757 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9066999999999999e-05 2023-10-20T03:36:03.757 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.757 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.758 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.758 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.376883+0000", 2023-10-20T03:36:03.758 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015189569 2023-10-20T03:36:03.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.758 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.758 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.759 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.376995+0000", 2023-10-20T03:36:03.759 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112099 2023-10-20T03:36:03.759 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.759 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.759 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.760 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.760 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.760 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.0s6 44/31 ECSubWrite(tid=19, reqid=client.4570.0:324, at_version=44'10, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:36:03.760 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.387754+0000", 2023-10-20T03:36:03.760 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 37.358323036999998, 2023-10-20T03:36:03.760 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020931727000000001, 2023-10-20T03:36:03.761 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.761 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.761 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.761 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.761 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.762 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387754+0000", 2023-10-20T03:36:03.762 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.762 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.762 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.762 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387754+0000", 2023-10-20T03:36:03.763 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.763 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.763 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387756+0000", 2023-10-20T03:36:03.764 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.573e-06 2023-10-20T03:36:03.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.764 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.764 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.764 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388025+0000", 2023-10-20T03:36:03.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026924199999999999 2023-10-20T03:36:03.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.765 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.765 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.765 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388058+0000", 2023-10-20T03:36:03.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2339999999999999e-05 2023-10-20T03:36:03.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.766 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.766 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.766 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388073+0000", 2023-10-20T03:36:03.766 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5489e-05 2023-10-20T03:36:03.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.767 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.767 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.767 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388242+0000", 2023-10-20T03:36:03.767 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016881200000000001 2023-10-20T03:36:03.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.768 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388268+0000", 2023-10-20T03:36:03.768 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6169999999999998e-05 2023-10-20T03:36:03.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.769 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.769 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.769 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.408608+0000", 2023-10-20T03:36:03.769 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020340272999999999 2023-10-20T03:36:03.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.769 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.770 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.770 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.408686+0000", 2023-10-20T03:36:03.770 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7828000000000001e-05 2023-10-20T03:36:03.770 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.770 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.771 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.771 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.771 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.0s6 49/31 ECSubWrite(tid=39, reqid=client.4570.0:390, at_version=49'23, trim_to=0'0, roll_forward_to=47'21))", 2023-10-20T03:36:03.771 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:31.413616+0000", 2023-10-20T03:36:03.772 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 32.332461743000003, 2023-10-20T03:36:03.772 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.02288652, 2023-10-20T03:36:03.772 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.772 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.772 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.773 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.773 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.413616+0000", 2023-10-20T03:36:03.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.774 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.413616+0000", 2023-10-20T03:36:03.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.413617+0000", 2023-10-20T03:36:03.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2600000000000002e-07 2023-10-20T03:36:03.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.776 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.414748+0000", 2023-10-20T03:36:03.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0011315520000000001 2023-10-20T03:36:03.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.776 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.776 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.414773+0000", 2023-10-20T03:36:03.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.478e-05 2023-10-20T03:36:03.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.414788+0000", 2023-10-20T03:36:03.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4926000000000001e-05 2023-10-20T03:36:03.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.415003+0000", 2023-10-20T03:36:03.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021510499999999999 2023-10-20T03:36:03.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.415029+0000", 2023-10-20T03:36:03.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6429999999999999e-05 2023-10-20T03:36:03.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.780 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.780 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.436443+0000", 2023-10-20T03:36:03.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021413835999999999 2023-10-20T03:36:03.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.781 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.781 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:31.436502+0000", 2023-10-20T03:36:03.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9165e-05 2023-10-20T03:36:03.781 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.782 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.782 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.782 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.3.22:2 3.as0 3.82f684ca (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:36:03.783 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:35.440266+0000", 2023-10-20T03:36:03.783 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 28.305811248000001, 2023-10-20T03:36:03.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.98271485700000005, 2023-10-20T03:36:03.783 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.783 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.784 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.784 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.784 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440266+0000", 2023-10-20T03:36:03.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.785 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.785 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440266+0000", 2023-10-20T03:36:03.785 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.786 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.786 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440269+0000", 2023-10-20T03:36:03.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4930000000000001e-06 2023-10-20T03:36:03.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.787 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.787 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440282+0000", 2023-10-20T03:36:03.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.33e-05 2023-10-20T03:36:03.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440310+0000", 2023-10-20T03:36:03.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7569999999999999e-05 2023-10-20T03:36:03.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440332+0000", 2023-10-20T03:36:03.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2633000000000002e-05 2023-10-20T03:36:03.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440473+0000", 2023-10-20T03:36:03.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014078499999999999 2023-10-20T03:36:03.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.790 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for peered", 2023-10-20T03:36:03.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:35.440478+0000", 2023-10-20T03:36:03.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3530000000000002e-06 2023-10-20T03:36:03.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.422959+0000", 2023-10-20T03:36:03.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.98248045299999998 2023-10-20T03:36:03.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.792 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:36.422981+0000", 2023-10-20T03:36:03.793 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2269999999999999e-05 2023-10-20T03:36:03.793 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.793 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.793 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.794 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:468 3.as0 3:51752c29:::smithi17039300-1:head [read 0~1327104,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-10-20T03:36:03.794 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.907631+0000", 2023-10-20T03:36:03.794 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.838446152, 2023-10-20T03:36:03.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016440132, 2023-10-20T03:36:03.794 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.795 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:03.795 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:03.795 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:03.795 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:03.795 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 468 2023-10-20T03:36:03.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.796 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907631+0000", 2023-10-20T03:36:03.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.797 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.797 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907631+0000", 2023-10-20T03:36:03.797 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.798 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.798 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907632+0000", 2023-10-20T03:36:03.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.048e-06 2023-10-20T03:36:03.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.799 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907644+0000", 2023-10-20T03:36:03.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1555e-05 2023-10-20T03:36:03.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907677+0000", 2023-10-20T03:36:03.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3207999999999998e-05 2023-10-20T03:36:03.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907691+0000", 2023-10-20T03:36:03.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3698e-05 2023-10-20T03:36:03.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.802 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.802 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.911974+0000", 2023-10-20T03:36:03.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0042835690000000001 2023-10-20T03:36:03.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.802 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:03.803 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.912070+0000", 2023-10-20T03:36:03.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5915999999999994e-05 2023-10-20T03:36:03.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.804 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.924072+0000", 2023-10-20T03:36:03.804 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012001138 2023-10-20T03:36:03.804 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.804 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.804 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.805 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:469 3.as0 3:51752c29:::smithi17039300-1:head [read 0~1327104,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2023-10-20T03:36:03.805 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.907719+0000", 2023-10-20T03:36:03.805 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.838358810999999, 2023-10-20T03:36:03.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018910997999999998, 2023-10-20T03:36:03.806 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.806 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:03.806 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:03.806 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:03.806 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:03.806 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 469 2023-10-20T03:36:03.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.807 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.807 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.807 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907719+0000", 2023-10-20T03:36:03.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907719+0000", 2023-10-20T03:36:03.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907720+0000", 2023-10-20T03:36:03.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.006e-06 2023-10-20T03:36:03.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.810 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.810 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.810 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907729+0000", 2023-10-20T03:36:03.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0520000000000007e-06 2023-10-20T03:36:03.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907743+0000", 2023-10-20T03:36:03.811 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4258e-05 2023-10-20T03:36:03.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.907757+0000", 2023-10-20T03:36:03.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3755e-05 2023-10-20T03:36:03.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.912479+0000", 2023-10-20T03:36:03.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0047223079999999997 2023-10-20T03:36:03.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:03.814 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.912642+0000", 2023-10-20T03:36:03.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000162795 2023-10-20T03:36:03.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.814 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.815 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.926630+0000", 2023-10-20T03:36:03.815 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013987823999999999 2023-10-20T03:36:03.815 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.815 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.815 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.816 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.0s6 69/31 ECSubWrite(tid=95, reqid=client.4570.0:552, at_version=69'41, trim_to=0'0, roll_forward_to=56'39))", 2023-10-20T03:36:03.816 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:50.954255+0000", 2023-10-20T03:36:03.816 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 12.791822742000001, 2023-10-20T03:36:03.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031876127999999997, 2023-10-20T03:36:03.817 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.817 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.817 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.817 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.818 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:50.954255+0000", 2023-10-20T03:36:03.818 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.818 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.818 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:50.954255+0000", 2023-10-20T03:36:03.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.819 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.819 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:50.954255+0000", 2023-10-20T03:36:03.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.13e-07 2023-10-20T03:36:03.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:50.955341+0000", 2023-10-20T03:36:03.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001085501 2023-10-20T03:36:03.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:50.955369+0000", 2023-10-20T03:36:03.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8374e-05 2023-10-20T03:36:03.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:50.955385+0000", 2023-10-20T03:36:03.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6143000000000001e-05 2023-10-20T03:36:03.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.823 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.823 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:50.955580+0000", 2023-10-20T03:36:03.823 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000194499 2023-10-20T03:36:03.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.824 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.824 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:50.955611+0000", 2023-10-20T03:36:03.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1028999999999997e-05 2023-10-20T03:36:03.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:50.986059+0000", 2023-10-20T03:36:03.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0304479 2023-10-20T03:36:03.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:50.986131+0000", 2023-10-20T03:36:03.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2169000000000006e-05 2023-10-20T03:36:03.826 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.826 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.827 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.827 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.9s5 70/31 ECSubWrite(tid=48, reqid=client.4570.0:567, at_version=70'36, trim_to=0'0, roll_forward_to=60'34))", 2023-10-20T03:36:03.827 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:52.045203+0000", 2023-10-20T03:36:03.827 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.700874925000001, 2023-10-20T03:36:03.828 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015867767000000001, 2023-10-20T03:36:03.828 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.828 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.828 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.828 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.829 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.829 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.045203+0000", 2023-10-20T03:36:03.829 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.829 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.830 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.045203+0000", 2023-10-20T03:36:03.830 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.830 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.831 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.045203+0000", 2023-10-20T03:36:03.831 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9299999999999998e-07 2023-10-20T03:36:03.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.831 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.831 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.045840+0000", 2023-10-20T03:36:03.832 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00063658200000000001 2023-10-20T03:36:03.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.832 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.832 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.045871+0000", 2023-10-20T03:36:03.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1199999999999999e-05 2023-10-20T03:36:03.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.045896+0000", 2023-10-20T03:36:03.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4669999999999999e-05 2023-10-20T03:36:03.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.834 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.046116+0000", 2023-10-20T03:36:03.834 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022008100000000001 2023-10-20T03:36:03.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.835 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.835 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.046137+0000", 2023-10-20T03:36:03.835 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.124e-05 2023-10-20T03:36:03.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.836 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.836 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.060955+0000", 2023-10-20T03:36:03.836 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014818225000000001 2023-10-20T03:36:03.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.837 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.061070+0000", 2023-10-20T03:36:03.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115176 2023-10-20T03:36:03.837 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.837 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.838 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.838 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.9s5 70/31 ECSubWrite(tid=49, reqid=client.4570.0:568, at_version=70'37, trim_to=0'0, roll_forward_to=60'34))", 2023-10-20T03:36:03.838 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:52.066937+0000", 2023-10-20T03:36:03.839 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.679140106, 2023-10-20T03:36:03.839 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.023253679999999999, 2023-10-20T03:36:03.839 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.839 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.839 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.840 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.840 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.840 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.066937+0000", 2023-10-20T03:36:03.840 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.840 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.841 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.841 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.066937+0000", 2023-10-20T03:36:03.841 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.841 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.066940+0000", 2023-10-20T03:36:03.842 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.17e-06 2023-10-20T03:36:03.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.067332+0000", 2023-10-20T03:36:03.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00039266000000000003 2023-10-20T03:36:03.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.844 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.067363+0000", 2023-10-20T03:36:03.844 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0363000000000001e-05 2023-10-20T03:36:03.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.844 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.844 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.067379+0000", 2023-10-20T03:36:03.845 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6177000000000001e-05 2023-10-20T03:36:03.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.845 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.067526+0000", 2023-10-20T03:36:03.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000147105 2023-10-20T03:36:03.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.846 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.846 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.067545+0000", 2023-10-20T03:36:03.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.863e-05 2023-10-20T03:36:03.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.090100+0000", 2023-10-20T03:36:03.847 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022555138999999998 2023-10-20T03:36:03.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.848 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.848 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.090191+0000", 2023-10-20T03:36:03.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1435999999999997e-05 2023-10-20T03:36:03.849 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.849 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.849 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.849 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:582 3.as0 3:53216f41:::smithi17039300-20:head [read 0~802816,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:03.850 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:52.989126+0000", 2023-10-20T03:36:03.850 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.756951812000001, 2023-10-20T03:36:03.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019006379, 2023-10-20T03:36:03.850 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.850 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:03.851 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:03.851 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:03.851 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:03.851 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 582 2023-10-20T03:36:03.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.852 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.989126+0000", 2023-10-20T03:36:03.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.989126+0000", 2023-10-20T03:36:03.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.989128+0000", 2023-10-20T03:36:03.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8360000000000001e-06 2023-10-20T03:36:03.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.855 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.855 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.989144+0000", 2023-10-20T03:36:03.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6886999999999999e-05 2023-10-20T03:36:03.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.989180+0000", 2023-10-20T03:36:03.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.557e-05 2023-10-20T03:36:03.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.857 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.989206+0000", 2023-10-20T03:36:03.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.633e-05 2023-10-20T03:36:03.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.858 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.002880+0000", 2023-10-20T03:36:03.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013674023 2023-10-20T03:36:03.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.858 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:03.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.003112+0000", 2023-10-20T03:36:03.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000232048 2023-10-20T03:36:03.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.008132+0000", 2023-10-20T03:36:03.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050196850000000003 2023-10-20T03:36:03.860 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.860 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.860 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.861 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:595 3.as0 3:5c818ca3:::smithi17039300-19:head [read 0~1654784,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:03.861 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.012346+0000", 2023-10-20T03:36:03.861 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.733731798000001, 2023-10-20T03:36:03.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019127495000000001, 2023-10-20T03:36:03.861 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.862 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:03.862 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:03.862 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:03.862 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:03.862 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 595 2023-10-20T03:36:03.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.863 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.863 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.863 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.012346+0000", 2023-10-20T03:36:03.863 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.012346+0000", 2023-10-20T03:36:03.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.012347+0000", 2023-10-20T03:36:03.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5969999999999999e-06 2023-10-20T03:36:03.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.012357+0000", 2023-10-20T03:36:03.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3959999999999995e-06 2023-10-20T03:36:03.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.012380+0000", 2023-10-20T03:36:03.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3637000000000001e-05 2023-10-20T03:36:03.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.012400+0000", 2023-10-20T03:36:03.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9843e-05 2023-10-20T03:36:03.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.013836+0000", 2023-10-20T03:36:03.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014355220000000001 2023-10-20T03:36:03.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:03.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.014361+0000", 2023-10-20T03:36:03.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052495999999999999 2023-10-20T03:36:03.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.871 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.031473+0000", 2023-10-20T03:36:03.871 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017112539999999999 2023-10-20T03:36:03.871 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.871 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.871 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.872 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:596 3.as0 3:5c818ca3:::smithi17039300-19:head [read 0~1654784,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:03.872 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.012428+0000", 2023-10-20T03:36:03.872 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.733649519, 2023-10-20T03:36:03.872 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022659619999999998, 2023-10-20T03:36:03.873 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.873 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:03.873 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:03.873 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:03.873 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:03.873 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 596 2023-10-20T03:36:03.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.874 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.874 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.874 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.874 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.012428+0000", 2023-10-20T03:36:03.875 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.875 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.875 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.875 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.012428+0000", 2023-10-20T03:36:03.875 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.876 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.876 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.876 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.012429+0000", 2023-10-20T03:36:03.876 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.274e-06 2023-10-20T03:36:03.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.877 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.877 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.877 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.012437+0000", 2023-10-20T03:36:03.877 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8730000000000007e-06 2023-10-20T03:36:03.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.878 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.878 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.878 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.012467+0000", 2023-10-20T03:36:03.878 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0300000000000001e-05 2023-10-20T03:36:03.878 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.879 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.879 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.879 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.012486+0000", 2023-10-20T03:36:03.879 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8113000000000001e-05 2023-10-20T03:36:03.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.880 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.880 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.015427+0000", 2023-10-20T03:36:03.880 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0029411110000000002 2023-10-20T03:36:03.880 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.881 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:03.881 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.016312+0000", 2023-10-20T03:36:03.881 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00088540000000000005 2023-10-20T03:36:03.881 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.881 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.882 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.882 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.035088+0000", 2023-10-20T03:36:03.882 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018775548999999999 2023-10-20T03:36:03.882 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.882 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.882 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.883 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.883 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:598 3.as0 3:53216f41:::smithi17039300-20:head [append 0~737280] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:03.883 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.209494+0000", 2023-10-20T03:36:03.883 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.536583602, 2023-10-20T03:36:03.884 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032508131000000003, 2023-10-20T03:36:03.884 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.884 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:03.884 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:03.884 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:03.884 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:03.885 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 598 2023-10-20T03:36:03.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.885 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.885 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.885 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.886 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.209494+0000", 2023-10-20T03:36:03.886 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.886 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.886 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.886 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.209494+0000", 2023-10-20T03:36:03.887 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.887 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.887 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.887 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.209496+0000", 2023-10-20T03:36:03.888 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.57e-06 2023-10-20T03:36:03.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.210574+0000", 2023-10-20T03:36:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001077445 2023-10-20T03:36:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.890 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.891 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.891 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.210613+0000", 2023-10-20T03:36:03.891 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9242999999999997e-05 2023-10-20T03:36:03.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.891 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.892 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.892 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.210629+0000", 2023-10-20T03:36:03.892 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5953000000000001e-05 2023-10-20T03:36:03.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.892 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.893 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.893 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.210757+0000", 2023-10-20T03:36:03.893 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127909 2023-10-20T03:36:03.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.893 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.893 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:03.894 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.210993+0000", 2023-10-20T03:36:03.894 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00023559500000000001 2023-10-20T03:36:03.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.894 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.894 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.895 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.212783+0000", 2023-10-20T03:36:03.895 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0017905799999999999 2023-10-20T03:36:03.895 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.895 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.895 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.895 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.241172+0000", 2023-10-20T03:36:03.896 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.028388452000000002 2023-10-20T03:36:03.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.896 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.896 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:36:03.896 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.241639+0000", 2023-10-20T03:36:03.896 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046703599999999999 2023-10-20T03:36:03.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.897 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.897 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.897 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.242002+0000", 2023-10-20T03:36:03.897 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036334799999999999 2023-10-20T03:36:03.898 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.898 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.898 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.898 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:599 3.as0 3:53216f41:::smithi17039300-20:head [append 0~557056] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:03.899 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.219210+0000", 2023-10-20T03:36:03.899 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.526867855999999, 2023-10-20T03:36:03.899 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.039492865000000002, 2023-10-20T03:36:03.899 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.899 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:03.900 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:03.900 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:03.900 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:03.900 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 599 2023-10-20T03:36:03.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.900 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.901 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.901 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.219210+0000", 2023-10-20T03:36:03.901 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.902 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.902 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.902 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.219210+0000", 2023-10-20T03:36:03.902 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.902 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.903 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.903 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.219212+0000", 2023-10-20T03:36:03.903 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4169999999999999e-06 2023-10-20T03:36:03.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.903 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.904 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.904 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.219916+0000", 2023-10-20T03:36:03.904 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00070391799999999999 2023-10-20T03:36:03.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.904 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.905 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.219950+0000", 2023-10-20T03:36:03.905 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3840000000000001e-05 2023-10-20T03:36:03.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.905 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.905 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.905 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.219966+0000", 2023-10-20T03:36:03.906 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6555999999999999e-05 2023-10-20T03:36:03.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.906 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.906 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.906 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.228320+0000", 2023-10-20T03:36:03.907 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0083534999999999998 2023-10-20T03:36:03.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.907 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:03.907 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.228690+0000", 2023-10-20T03:36:03.907 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00037058399999999997 2023-10-20T03:36:03.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.908 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.908 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.908 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.230826+0000", 2023-10-20T03:36:03.908 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002135124 2023-10-20T03:36:03.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.909 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.909 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.257428+0000", 2023-10-20T03:36:03.909 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.026602276000000001 2023-10-20T03:36:03.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.910 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.910 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:36:03.910 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.257994+0000", 2023-10-20T03:36:03.910 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00056607499999999998 2023-10-20T03:36:03.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.911 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.911 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.258703+0000", 2023-10-20T03:36:03.911 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00070857500000000003 2023-10-20T03:36:03.911 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.912 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.912 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.912 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.912 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWriteReply(3.as0 71/56 ECSubWriteReply(tid=21, last_complete=71'27, committed=1, applied=1))", 2023-10-20T03:36:03.912 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.221879+0000", 2023-10-20T03:36:03.913 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.524198626, 2023-10-20T03:36:03.913 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019093038999999999, 2023-10-20T03:36:03.913 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.913 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.913 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.914 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.914 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.221879+0000", 2023-10-20T03:36:03.914 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.915 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.915 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.221879+0000", 2023-10-20T03:36:03.915 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.915 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.915 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.916 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.221881+0000", 2023-10-20T03:36:03.916 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9939999999999999e-06 2023-10-20T03:36:03.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.916 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.917 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.221892+0000", 2023-10-20T03:36:03.917 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0995999999999999e-05 2023-10-20T03:36:03.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.917 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.917 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.917 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.221921+0000", 2023-10-20T03:36:03.918 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9577000000000002e-05 2023-10-20T03:36:03.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.918 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.918 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.221939+0000", 2023-10-20T03:36:03.918 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7638999999999998e-05 2023-10-20T03:36:03.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.919 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.919 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.240931+0000", 2023-10-20T03:36:03.919 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018991563999999999 2023-10-20T03:36:03.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.920 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.920 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.920 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.240972+0000", 2023-10-20T03:36:03.920 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1269000000000001e-05 2023-10-20T03:36:03.920 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.921 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.921 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.921 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWriteReply(3.as0 71/56 ECSubWriteReply(tid=21, last_complete=71'27, committed=1, applied=1))", 2023-10-20T03:36:03.922 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.221897+0000", 2023-10-20T03:36:03.922 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.524180786000001, 2023-10-20T03:36:03.922 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019278264, 2023-10-20T03:36:03.922 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.922 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.922 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.923 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.923 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.221897+0000", 2023-10-20T03:36:03.923 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.924 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.924 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.221897+0000", 2023-10-20T03:36:03.924 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.925 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.925 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.221899+0000", 2023-10-20T03:36:03.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8899999999999999e-06 2023-10-20T03:36:03.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.926 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.926 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.221909+0000", 2023-10-20T03:36:03.926 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0594000000000001e-05 2023-10-20T03:36:03.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.926 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.927 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.221942+0000", 2023-10-20T03:36:03.927 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2356e-05 2023-10-20T03:36:03.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.927 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.927 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.927 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.221965+0000", 2023-10-20T03:36:03.928 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3085999999999999e-05 2023-10-20T03:36:03.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.928 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.928 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.241092+0000", 2023-10-20T03:36:03.929 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019127509000000001 2023-10-20T03:36:03.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.929 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.929 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.241175+0000", 2023-10-20T03:36:03.930 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2829000000000003e-05 2023-10-20T03:36:03.930 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.930 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.930 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.930 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.931 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:600 3.as0 3:53216f41:::smithi17039300-20:head [append 0~606208] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:03.931 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.231251+0000", 2023-10-20T03:36:03.931 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.51482698, 2023-10-20T03:36:03.931 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.028703586, 2023-10-20T03:36:03.931 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.932 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:03.932 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:03.932 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:03.932 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:03.932 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 600 2023-10-20T03:36:03.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.933 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.933 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.231251+0000", 2023-10-20T03:36:03.933 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.934 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.934 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.231251+0000", 2023-10-20T03:36:03.934 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.935 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.935 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.231254+0000", 2023-10-20T03:36:03.935 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3900000000000002e-06 2023-10-20T03:36:03.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.936 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232334+0000", 2023-10-20T03:36:03.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0010801649999999999 2023-10-20T03:36:03.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.937 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232366+0000", 2023-10-20T03:36:03.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2227000000000002e-05 2023-10-20T03:36:03.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.938 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.938 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232381+0000", 2023-10-20T03:36:03.938 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4936000000000001e-05 2023-10-20T03:36:03.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.938 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.939 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.242571+0000", 2023-10-20T03:36:03.939 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010189969 2023-10-20T03:36:03.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.939 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:03.939 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.242816+0000", 2023-10-20T03:36:03.940 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000245262 2023-10-20T03:36:03.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.940 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.940 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.244327+0000", 2023-10-20T03:36:03.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0015105990000000001 2023-10-20T03:36:03.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.258736+0000", 2023-10-20T03:36:03.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014408591 2023-10-20T03:36:03.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.942 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:36:03.942 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.259567+0000", 2023-10-20T03:36:03.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00083142999999999997 2023-10-20T03:36:03.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.943 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.943 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.259954+0000", 2023-10-20T03:36:03.943 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038701700000000003 2023-10-20T03:36:03.943 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.944 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.944 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.944 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:601 3.as0 3:53216f41:::smithi17039300-20:head [setxattr _header (57) in=7b] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:03.945 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.232411+0000", 2023-10-20T03:36:03.945 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.513666426, 2023-10-20T03:36:03.945 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.043748984999999997, 2023-10-20T03:36:03.945 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.945 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:03.945 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:03.946 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:03.946 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:03.946 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 601 2023-10-20T03:36:03.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.946 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.947 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.947 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232411+0000", 2023-10-20T03:36:03.947 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.948 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.948 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232411+0000", 2023-10-20T03:36:03.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.948 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.949 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232412+0000", 2023-10-20T03:36:03.949 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2500000000000001e-06 2023-10-20T03:36:03.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.949 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.949 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232418+0000", 2023-10-20T03:36:03.950 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5740000000000003e-06 2023-10-20T03:36:03.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.950 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.950 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232434+0000", 2023-10-20T03:36:03.951 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5926e-05 2023-10-20T03:36:03.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.951 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.951 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.951 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232446+0000", 2023-10-20T03:36:03.952 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2327000000000001e-05 2023-10-20T03:36:03.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.952 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.952 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.952 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.254971+0000", 2023-10-20T03:36:03.952 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022525318999999999 2023-10-20T03:36:03.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.953 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.953 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:03.953 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.255309+0000", 2023-10-20T03:36:03.953 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000337361 2023-10-20T03:36:03.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.954 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.954 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.954 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.256766+0000", 2023-10-20T03:36:03.954 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001456653 2023-10-20T03:36:03.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.955 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.955 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.955 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.259382+0000", 2023-10-20T03:36:03.955 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026165799999999999 2023-10-20T03:36:03.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.955 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.956 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:36:03.956 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.275760+0000", 2023-10-20T03:36:03.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016377458000000001 2023-10-20T03:36:03.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.957 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.957 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.276160+0000", 2023-10-20T03:36:03.957 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040053700000000002 2023-10-20T03:36:03.957 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.957 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.957 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.958 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:602 3.as0 3:53216f41:::smithi17039300-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:03.958 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.232461+0000", 2023-10-20T03:36:03.958 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.513616979, 2023-10-20T03:36:03.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.046646883, 2023-10-20T03:36:03.959 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.959 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:03.959 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:03.959 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:03.959 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:03.960 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 602 2023-10-20T03:36:03.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.960 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.960 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.960 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.960 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232461+0000", 2023-10-20T03:36:03.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.961 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232461+0000", 2023-10-20T03:36:03.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.962 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232461+0000", 2023-10-20T03:36:03.962 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5300000000000003e-07 2023-10-20T03:36:03.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.963 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.963 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.963 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232465+0000", 2023-10-20T03:36:03.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4539999999999999e-06 2023-10-20T03:36:03.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.964 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.964 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.964 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232479+0000", 2023-10-20T03:36:03.964 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.398e-05 2023-10-20T03:36:03.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.965 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.965 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.232487+0000", 2023-10-20T03:36:03.965 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0290000000000005e-06 2023-10-20T03:36:03.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.260464+0000", 2023-10-20T03:36:03.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027977512 2023-10-20T03:36:03.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:03.967 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.260637+0000", 2023-10-20T03:36:03.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017235500000000001 2023-10-20T03:36:03.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.967 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.968 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.276885+0000", 2023-10-20T03:36:03.968 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016248296999999998 2023-10-20T03:36:03.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.968 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.968 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:03.968 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.277058+0000", 2023-10-20T03:36:03.969 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173165 2023-10-20T03:36:03.969 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.969 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.969 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.969 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.279107+0000", 2023-10-20T03:36:03.970 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0020492380000000001 2023-10-20T03:36:03.970 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.970 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.970 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.971 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.0s6 75/31 ECSubWrite(tid=97, reqid=client.4570.0:628, at_version=75'43, trim_to=0'0, roll_forward_to=69'41))", 2023-10-20T03:36:03.971 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:57.036511+0000", 2023-10-20T03:36:03.971 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.7095662310000002, 2023-10-20T03:36:03.971 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041192095999999997, 2023-10-20T03:36:03.971 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:03.972 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:03.972 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:03.972 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.972 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:03.972 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.036511+0000", 2023-10-20T03:36:03.972 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.973 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.973 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:03.973 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.036511+0000", 2023-10-20T03:36:03.973 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:03.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.036512+0000", 2023-10-20T03:36:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.13e-07 2023-10-20T03:36:03.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.975 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.975 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:03.975 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.038649+0000", 2023-10-20T03:36:03.975 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0021368670000000002 2023-10-20T03:36:03.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.975 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.976 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:03.976 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.038693+0000", 2023-10-20T03:36:03.976 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4709999999999997e-05 2023-10-20T03:36:03.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:03.977 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.038720+0000", 2023-10-20T03:36:03.977 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6546e-05 2023-10-20T03:36:03.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.977 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:03.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.038941+0000", 2023-10-20T03:36:03.978 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000221249 2023-10-20T03:36:03.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.978 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.978 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:03.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.038963+0000", 2023-10-20T03:36:03.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1872e-05 2023-10-20T03:36:03.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:03.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.077616+0000", 2023-10-20T03:36:03.980 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038652736 2023-10-20T03:36:03.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:03.980 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:03.980 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:03.980 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.077703+0000", 2023-10-20T03:36:03.980 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7503000000000006e-05 2023-10-20T03:36:03.981 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.981 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.981 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.981 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:03.981 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:03.982 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:03.983 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:36:03.984 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:36:03.985 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:36:03.985 INFO:tasks.rados.rados.0.smithi170.stdout:273: done (2 left) 2023-10-20T03:36:03.985 INFO:tasks.rados.rados.0.smithi170.stdout:274: done (1 left) 2023-10-20T03:36:03.985 INFO:tasks.rados.rados.0.smithi170.stdout:275: done (0 left) 2023-10-20T03:36:03.986 INFO:tasks.rados.rados.0.smithi170.stdout:276: rollback oid 22 current snap is 24 2023-10-20T03:36:03.986 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 22 to 23 2023-10-20T03:36:03.986 INFO:tasks.rados.rados.0.smithi170.stdout:277: snap_create 2023-10-20T03:36:03.986 INFO:tasks.rados.rados.0.smithi170.stdout:276: finishing rollback tid 0 to smithi17039300-22 2023-10-20T03:36:03.986 INFO:tasks.rados.rados.0.smithi170.stdout:276: finishing rollback tid 1 to smithi17039300-22 2023-10-20T03:36:03.987 INFO:tasks.rados.rados.0.smithi170.stdout:276: finishing rollback tid 2 to smithi17039300-22 2023-10-20T03:36:03.987 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 39 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-10-20T03:36:03.987 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-10-20T03:36:03.987 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-10-20T03:36:03.988 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:36:03.988 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:36:03.989 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:03.980+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:04.035 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:36:04.035 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:36:04.072 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:36:04.073 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:36:04.084 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:36:04.083+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:36:04.107 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:36:04.124 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:04.124 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:04.139 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-10-20T03:36:04.140 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:36:04.141 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-10-20T03:36:04.142 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:36:04.136+0000 7f6c2e7fc640 0 -- 172.21.15.33:0/4018636134 >> [v2:172.21.15.170:6819/826332016,v1:172.21.15.170:6822/826332016] conn(0x7f6c3021a3d0 msgr2=0x7f6c3021a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-10-20T03:36:04.185 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:36:04.183+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:36:04.265 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:04.266 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:04.279 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-10-20T03:36:04.286 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:36:04.283+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:36:04.375 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:36:04.375 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:36:04.381 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:36:04.382 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:36:04.387 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:04.384+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:04.409 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:04.410 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:04.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-10-20T03:36:04.488 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:04.484+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:04.558 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:04.558 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:04.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-10-20T03:36:04.589 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:04.588+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:04.690 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:04.690 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:04.690 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:04.688+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:04.704 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-10-20T03:36:04.784 INFO:tasks.rados.rados.0.smithi170.stdout:276: done (1 left) 2023-10-20T03:36:04.785 INFO:tasks.rados.rados.0.smithi170.stdout:277: done (0 left) 2023-10-20T03:36:04.785 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 19 current snap is 25 2023-10-20T03:36:04.785 INFO:tasks.rados.rados.0.smithi170.stdout:278: seq_num 120 ranges {1654784=475136,2129920=655360,2785280=622592,3407872=278528} 2023-10-20T03:36:04.792 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:04.788+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:04.802 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-10-20T03:36:04.812 INFO:tasks.rados.rados.0.smithi170.stdout:278: writing smithi17039300-19 from 1654784 to 2129920 tid 1 2023-10-20T03:36:04.821 INFO:tasks.rados.rados.0.smithi170.stdout:278: writing smithi17039300-19 from 2129920 to 2785280 tid 2 2023-10-20T03:36:04.824 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:04.824 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:04.830 INFO:tasks.rados.rados.0.smithi170.stdout:278: writing smithi17039300-19 from 2785280 to 3407872 tid 3 2023-10-20T03:36:04.834 INFO:tasks.rados.rados.0.smithi170.stdout:278: writing smithi17039300-19 from 3407872 to 3686400 tid 4 2023-10-20T03:36:04.835 INFO:tasks.rados.rados.0.smithi170.stdout:279: rollback oid 42 current snap is 25 2023-10-20T03:36:04.835 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 42 to 24 2023-10-20T03:36:04.835 INFO:tasks.rados.rados.0.smithi170.stdout:280: snap_remove snap 23 2023-10-20T03:36:04.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-10-20T03:36:04.893 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:04.889+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:04.963 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:04.963 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:04.977 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-10-20T03:36:04.994 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:36:04.988+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:36:05.095 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:05.092+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:05.096 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:05.096 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:05.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-10-20T03:36:05.159 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15] out_osds: [12] dead_osds: [6] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 10, 14, 3, 7, 11, 15] 2023-10-20T03:36:05.160 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-10-20T03:36:05.160 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2023-10-20T03:36:05.160 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-20T03:36:05.161 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-20T03:36:05.163 INFO:tasks.ceph.osd.6:Started 2023-10-20T03:36:05.164 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-10-20T03:36:05.196 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:05.192+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:05.237 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:05.238 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:05.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-10-20T03:36:05.283 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2023-10-20T03:36:05.284 INFO:teuthology.orchestra.run.smithi064.stderr:0 2023-10-20T03:36:05.284 INFO:teuthology.orchestra.run.smithi064.stderr:1 2023-10-20T03:36:05.284 INFO:teuthology.orchestra.run.smithi064.stderr:2 2023-10-20T03:36:05.284 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2023-10-20T03:36:05.284 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2023-10-20T03:36:05.284 INFO:teuthology.orchestra.run.smithi064.stderr:config diff 2023-10-20T03:36:05.285 INFO:teuthology.orchestra.run.smithi064.stderr:config diff get 2023-10-20T03:36:05.285 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2023-10-20T03:36:05.285 INFO:teuthology.orchestra.run.smithi064.stderr:config help [] 2023-10-20T03:36:05.285 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2023-10-20T03:36:05.285 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2023-10-20T03:36:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-20T03:36:05.287 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2023-10-20T03:36:05.332 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:05.331+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:05.368 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:05.368 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:05.382 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-10-20T03:36:05.398 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:05.393+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:05.499 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:05.496+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:05.552 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:05.552 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:05.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-10-20T03:36:05.599 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:36:05.596+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:36:05.690 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:05.691 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:05.700 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:05.699+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:05.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-10-20T03:36:05.781 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2023-10-20T03:36:05.785 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:05.783+0000 7f7215348640 -1 Falling back to public interface 2023-10-20T03:36:05.793 INFO:tasks.rados.rados.0.smithi170.stdout:280: done (2 left) 2023-10-20T03:36:05.793 INFO:tasks.rados.rados.0.smithi170.stdout:281: read oid 4 snap -1 2023-10-20T03:36:05.793 INFO:tasks.rados.rados.0.smithi170.stdout:281: expect (ObjNum 103 snap 13 seq_num 103) 2023-10-20T03:36:05.793 INFO:tasks.rados.rados.0.smithi170.stdout:279: finishing rollback tid 0 to smithi17039300-42 2023-10-20T03:36:05.794 INFO:tasks.rados.rados.0.smithi170.stdout:279: finishing rollback tid 1 to smithi17039300-42 2023-10-20T03:36:05.794 INFO:tasks.rados.rados.0.smithi170.stdout:279: finishing rollback tid 2 to smithi17039300-42 2023-10-20T03:36:05.794 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 42 v 58 (ObjNum 60 snap 2 seq_num 60) dirty exists 2023-10-20T03:36:05.794 INFO:tasks.rados.rados.0.smithi170.stdout:278: finishing write tid 1 to smithi17039300-19 2023-10-20T03:36:05.794 INFO:tasks.rados.rados.0.smithi170.stdout:278: finishing write tid 2 to smithi17039300-19 2023-10-20T03:36:05.795 INFO:tasks.rados.rados.0.smithi170.stdout:278: finishing write tid 3 to smithi17039300-19 2023-10-20T03:36:05.795 INFO:tasks.rados.rados.0.smithi170.stdout:278: finishing write tid 4 to smithi17039300-19 2023-10-20T03:36:05.795 INFO:tasks.rados.rados.0.smithi170.stdout:278: finishing write tid 5 to smithi17039300-19 2023-10-20T03:36:05.795 INFO:tasks.rados.rados.0.smithi170.stdout:278: finishing write tid 6 to smithi17039300-19 2023-10-20T03:36:05.795 INFO:tasks.rados.rados.0.smithi170.stdout:278: oid 19 updating version 0 to 44 2023-10-20T03:36:05.796 INFO:tasks.rados.rados.0.smithi170.stdout:278: oid 19 version 44 is already newer than 42 2023-10-20T03:36:05.796 INFO:tasks.rados.rados.0.smithi170.stdout:278: oid 19 version 44 is already newer than 43 2023-10-20T03:36:05.796 INFO:tasks.rados.rados.0.smithi170.stdout:278: oid 19 updating version 44 to 45 2023-10-20T03:36:05.796 INFO:tasks.rados.rados.0.smithi170.stdout:278: oid 19 version 45 is already newer than 41 2023-10-20T03:36:05.796 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 45 (ObjNum 120 snap 25 seq_num 120) dirty exists 2023-10-20T03:36:05.797 INFO:tasks.rados.rados.0.smithi170.stdout:278: left oid 19 (ObjNum 120 snap 25 seq_num 120) 2023-10-20T03:36:05.797 INFO:tasks.rados.rados.0.smithi170.stdout:282: read oid 7 snap 24 2023-10-20T03:36:05.797 INFO:tasks.rados.rados.0.smithi170.stdout:282: expect (ObjNum 66 snap 4 seq_num 66) 2023-10-20T03:36:05.797 INFO:tasks.rados.rados.0.smithi170.stdout:278: done (3 left) 2023-10-20T03:36:05.797 INFO:tasks.rados.rados.0.smithi170.stdout:279: done (2 left) 2023-10-20T03:36:05.798 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 5 current snap is 25 2023-10-20T03:36:05.798 INFO:tasks.rados.rados.0.smithi170.stdout:283: seq_num 121 ranges {1146880=491520,1638400=802816,2441216=720896,3162112=32768} 2023-10-20T03:36:05.801 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:05.796+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:05.813 INFO:tasks.rados.rados.0.smithi170.stdout:283: writing smithi17039300-5 from 1146880 to 1638400 tid 1 2023-10-20T03:36:05.824 INFO:tasks.rados.rados.0.smithi170.stdout:283: writing smithi17039300-5 from 1638400 to 2441216 tid 2 2023-10-20T03:36:05.833 INFO:tasks.rados.rados.0.smithi170.stdout:283: writing smithi17039300-5 from 2441216 to 3162112 tid 3 2023-10-20T03:36:05.833 INFO:tasks.rados.rados.0.smithi170.stdout:283: writing smithi17039300-5 from 3162112 to 3194880 tid 4 2023-10-20T03:36:05.834 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 50 current snap is 25 2023-10-20T03:36:05.834 INFO:tasks.rados.rados.0.smithi170.stdout:284: seq_num 122 ranges {4259840=507904,4767744=425984,5193728=442368,5636096=540672,6176768=147456} 2023-10-20T03:36:05.850 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:05.850 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:05.866 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-10-20T03:36:05.871 INFO:tasks.rados.rados.0.smithi170.stdout:284: writing smithi17039300-50 from 4259840 to 4767744 tid 1 2023-10-20T03:36:05.874 INFO:tasks.rados.rados.0.smithi170.stdout:284: writing smithi17039300-50 from 4767744 to 5193728 tid 2 2023-10-20T03:36:05.877 INFO:tasks.rados.rados.0.smithi170.stdout:284: writing smithi17039300-50 from 5193728 to 5636096 tid 3 2023-10-20T03:36:05.881 INFO:tasks.rados.rados.0.smithi170.stdout:284: writing smithi17039300-50 from 5636096 to 6176768 tid 4 2023-10-20T03:36:05.882 INFO:tasks.rados.rados.0.smithi170.stdout:284: writing smithi17039300-50 from 6176768 to 6324224 tid 5 2023-10-20T03:36:05.882 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 35 current snap is 25 2023-10-20T03:36:05.882 INFO:tasks.rados.rados.0.smithi170.stdout:282: expect (ObjNum 66 snap 4 seq_num 66) 2023-10-20T03:36:05.901 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:05.896+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:05.937 INFO:tasks.rados.rados.0.smithi170.stdout:281: expect (ObjNum 103 snap 13 seq_num 103) 2023-10-20T03:36:06.002 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:06.000+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:06.003 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:06.003 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:06.018 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-10-20T03:36:06.072 INFO:tasks.rados.rados.0.smithi170.stdout:283: finishing write tid 1 to smithi17039300-5 2023-10-20T03:36:06.072 INFO:tasks.rados.rados.0.smithi170.stdout:283: finishing write tid 2 to smithi17039300-5 2023-10-20T03:36:06.073 INFO:tasks.rados.rados.0.smithi170.stdout:283: finishing write tid 3 to smithi17039300-5 2023-10-20T03:36:06.073 INFO:tasks.rados.rados.0.smithi170.stdout:283: finishing write tid 4 to smithi17039300-5 2023-10-20T03:36:06.073 INFO:tasks.rados.rados.0.smithi170.stdout:283: finishing write tid 5 to smithi17039300-5 2023-10-20T03:36:06.073 INFO:tasks.rados.rados.0.smithi170.stdout:283: finishing write tid 6 to smithi17039300-5 2023-10-20T03:36:06.073 INFO:tasks.rados.rados.0.smithi170.stdout:283: oid 5 updating version 0 to 30 2023-10-20T03:36:06.074 INFO:tasks.rados.rados.0.smithi170.stdout:283: oid 5 updating version 30 to 34 2023-10-20T03:36:06.074 INFO:tasks.rados.rados.0.smithi170.stdout:283: oid 5 version 34 is already newer than 33 2023-10-20T03:36:06.074 INFO:tasks.rados.rados.0.smithi170.stdout:283: oid 5 version 34 is already newer than 31 2023-10-20T03:36:06.074 INFO:tasks.rados.rados.0.smithi170.stdout:283: oid 5 version 34 is already newer than 32 2023-10-20T03:36:06.075 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 34 (ObjNum 121 snap 25 seq_num 121) dirty exists 2023-10-20T03:36:06.075 INFO:tasks.rados.rados.0.smithi170.stdout:283: left oid 5 (ObjNum 121 snap 25 seq_num 121) 2023-10-20T03:36:06.075 INFO:tasks.rados.rados.0.smithi170.stdout:284: finishing write tid 1 to smithi17039300-50 2023-10-20T03:36:06.075 INFO:tasks.rados.rados.0.smithi170.stdout:284: finishing write tid 2 to smithi17039300-50 2023-10-20T03:36:06.075 INFO:tasks.rados.rados.0.smithi170.stdout:284: finishing write tid 3 to smithi17039300-50 2023-10-20T03:36:06.076 INFO:tasks.rados.rados.0.smithi170.stdout:284: finishing write tid 4 to smithi17039300-50 2023-10-20T03:36:06.076 INFO:tasks.rados.rados.0.smithi170.stdout:284: finishing write tid 5 to smithi17039300-50 2023-10-20T03:36:06.076 INFO:tasks.rados.rados.0.smithi170.stdout:284: finishing write tid 6 to smithi17039300-50 2023-10-20T03:36:06.076 INFO:tasks.rados.rados.0.smithi170.stdout:284: finishing write tid 7 to smithi17039300-50 2023-10-20T03:36:06.076 INFO:tasks.rados.rados.0.smithi170.stdout:284: oid 50 updating version 0 to 56 2023-10-20T03:36:06.077 INFO:tasks.rados.rados.0.smithi170.stdout:284: oid 50 updating version 56 to 58 2023-10-20T03:36:06.077 INFO:tasks.rados.rados.0.smithi170.stdout:284: oid 50 updating version 58 to 60 2023-10-20T03:36:06.077 INFO:tasks.rados.rados.0.smithi170.stdout:284: oid 50 updating version 60 to 61 2023-10-20T03:36:06.077 INFO:tasks.rados.rados.0.smithi170.stdout:284: oid 50 version 61 is already newer than 57 2023-10-20T03:36:06.077 INFO:tasks.rados.rados.0.smithi170.stdout:284: oid 50 version 61 is already newer than 59 2023-10-20T03:36:06.078 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 50 v 61 (ObjNum 122 snap 25 seq_num 122) dirty exists 2023-10-20T03:36:06.078 INFO:tasks.rados.rados.0.smithi170.stdout:284: left oid 50 (ObjNum 122 snap 25 seq_num 122) 2023-10-20T03:36:06.078 INFO:tasks.rados.rados.0.smithi170.stdout:285: seq_num 123 ranges {573440=524288,1097728=573440,1671168=688128,2359296=294912} 2023-10-20T03:36:06.080 INFO:tasks.rados.rados.0.smithi170.stdout:285: writing smithi17039300-35 from 573440 to 1097728 tid 1 2023-10-20T03:36:06.084 INFO:tasks.rados.rados.0.smithi170.stdout:285: writing smithi17039300-35 from 1097728 to 1671168 tid 2 2023-10-20T03:36:06.090 INFO:tasks.rados.rados.0.smithi170.stdout:285: writing smithi17039300-35 from 1671168 to 2359296 tid 3 2023-10-20T03:36:06.092 INFO:tasks.rados.rados.0.smithi170.stdout:285: writing smithi17039300-35 from 2359296 to 2654208 tid 4 2023-10-20T03:36:06.093 INFO:tasks.rados.rados.0.smithi170.stdout:281: done (4 left) 2023-10-20T03:36:06.093 INFO:tasks.rados.rados.0.smithi170.stdout:282: done (3 left) 2023-10-20T03:36:06.093 INFO:tasks.rados.rados.0.smithi170.stdout:283: done (2 left) 2023-10-20T03:36:06.093 INFO:tasks.rados.rados.0.smithi170.stdout:284: done (1 left) 2023-10-20T03:36:06.094 INFO:tasks.rados.rados.0.smithi170.stdout:286: rollback oid 18 current snap is 25 2023-10-20T03:36:06.094 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 18 to 24 2023-10-20T03:36:06.094 INFO:tasks.rados.rados.0.smithi170.stdout:287: delete oid 24 current snap is 25 2023-10-20T03:36:06.094 INFO:tasks.rados.rados.0.smithi170.stdout:286: finishing rollback tid 1 to smithi17039300-18 2023-10-20T03:36:06.094 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 18 v 0 (ObjNum 81 snap 0 seq_num 0) dirty dne 2023-10-20T03:36:06.098 INFO:tasks.rados.rados.0.smithi170.stdout:286: done (2 left) 2023-10-20T03:36:06.098 INFO:tasks.rados.rados.0.smithi170.stdout:287: done (1 left) 2023-10-20T03:36:06.098 INFO:tasks.rados.rados.0.smithi170.stdout:288: snap_create 2023-10-20T03:36:06.102 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:06.097+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:06.117 INFO:tasks.rados.rados.0.smithi170.stdout:285: finishing write tid 1 to smithi17039300-35 2023-10-20T03:36:06.147 INFO:tasks.rados.rados.0.smithi170.stdout:285: finishing write tid 2 to smithi17039300-35 2023-10-20T03:36:06.148 INFO:tasks.rados.rados.0.smithi170.stdout:285: finishing write tid 3 to smithi17039300-35 2023-10-20T03:36:06.152 INFO:tasks.rados.rados.0.smithi170.stdout:285: finishing write tid 4 to smithi17039300-35 2023-10-20T03:36:06.154 INFO:tasks.rados.rados.0.smithi170.stdout:285: finishing write tid 5 to smithi17039300-35 2023-10-20T03:36:06.159 INFO:tasks.rados.rados.0.smithi170.stdout:285: finishing write tid 6 to smithi17039300-35 2023-10-20T03:36:06.159 INFO:tasks.rados.rados.0.smithi170.stdout:285: oid 35 updating version 0 to 62 2023-10-20T03:36:06.159 INFO:tasks.rados.rados.0.smithi170.stdout:285: oid 35 version 62 is already newer than 58 2023-10-20T03:36:06.159 INFO:tasks.rados.rados.0.smithi170.stdout:285: oid 35 version 62 is already newer than 61 2023-10-20T03:36:06.160 INFO:tasks.rados.rados.0.smithi170.stdout:285: oid 35 version 62 is already newer than 59 2023-10-20T03:36:06.160 INFO:tasks.rados.rados.0.smithi170.stdout:285: oid 35 version 62 is already newer than 60 2023-10-20T03:36:06.160 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 62 (ObjNum 123 snap 25 seq_num 123) dirty exists 2023-10-20T03:36:06.160 INFO:tasks.rados.rados.0.smithi170.stdout:285: left oid 35 (ObjNum 123 snap 25 seq_num 123) 2023-10-20T03:36:06.181 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:06.181 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:06.195 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-10-20T03:36:06.204 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:36:06.200+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:36:06.305 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:06.300+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:06.358 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:06.359 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:06.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-10-20T03:36:06.383 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:36:06.430 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:06.428+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:06.489 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:06.489 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:06.503 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-10-20T03:36:06.506 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:06.504+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:06.543 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:36:06.544 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.564 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:36:06.565 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.585 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:36:06.585 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.603 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:36:06.603 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.606 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:06.604+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:06.620 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:36:06.621 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.627 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:06.627 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:06.641 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:36:06.641 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.642 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:36:06.643 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-10-20T03:36:06.662 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:36:06.662 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.680 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:36:06.681 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.703 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:36:06.704 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.708 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:06.704+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:06.724 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:36:06.725 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.745 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:36:06.745 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.762 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:36:06.762 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.765 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:06.765 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:36:06.765 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2023-10-20T03:36:06.765 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:06.779 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-10-20T03:36:06.782 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:36:06.783 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.803 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:36:06.803 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.808 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:06.804+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:06.820 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:36:06.820 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:36:06.825 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:36:06.826 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:36:06.897 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:06.898 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:36:06.898 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:36:06.898 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:36:06.898 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:06.909 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:06.904+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:06.911 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-10-20T03:36:07.010 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:36:07.008+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:36:07.035 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:07.036 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2023-10-20T03:36:07.036 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2023-10-20T03:36:07.036 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2023-10-20T03:36:07.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.036 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:208 3.es0 3:7dffa048:::smithi17039300-5:head [append 0~688128] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:36:07.037 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.269565+0000", 2023-10-20T03:36:07.037 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 47.762882036999997, 2023-10-20T03:36:07.037 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029923571, 2023-10-20T03:36:07.037 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.037 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:07.037 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:07.038 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:07.038 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:07.038 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 208 2023-10-20T03:36:07.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.038 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.039 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.039 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.039 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.269565+0000", 2023-10-20T03:36:07.039 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.040 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.040 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.040 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.269565+0000", 2023-10-20T03:36:07.040 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.040 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.041 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.041 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.041 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.269569+0000", 2023-10-20T03:36:07.041 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0230000000000001e-06 2023-10-20T03:36:07.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.042 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.042 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.042 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270337+0000", 2023-10-20T03:36:07.042 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00076892900000000001 2023-10-20T03:36:07.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.043 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.043 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.043 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270372+0000", 2023-10-20T03:36:07.043 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4652e-05 2023-10-20T03:36:07.043 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.043 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.044 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.044 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270391+0000", 2023-10-20T03:36:07.044 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8819000000000001e-05 2023-10-20T03:36:07.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.044 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.045 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.045 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270555+0000", 2023-10-20T03:36:07.045 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016383700000000001 2023-10-20T03:36:07.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.045 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.046 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:07.046 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270730+0000", 2023-10-20T03:36:07.046 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017536 2023-10-20T03:36:07.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.046 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.047 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.047 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.272967+0000", 2023-10-20T03:36:07.047 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002236601 2023-10-20T03:36:07.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.047 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.047 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.048 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298706+0000", 2023-10-20T03:36:07.048 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025739482000000001 2023-10-20T03:36:07.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.048 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:07.049 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299042+0000", 2023-10-20T03:36:07.049 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00033608100000000001 2023-10-20T03:36:07.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.049 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.049 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.050 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299489+0000", 2023-10-20T03:36:07.050 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00044678699999999998 2023-10-20T03:36:07.050 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.050 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.050 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.051 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.051 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:209 3.es0 3:7dffa048:::smithi17039300-5:head [append 0~376832] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:36:07.051 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.277974+0000", 2023-10-20T03:36:07.051 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 47.754473202, 2023-10-20T03:36:07.051 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.023138072999999999, 2023-10-20T03:36:07.052 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.052 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:07.052 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:07.052 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:07.052 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:07.053 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 209 2023-10-20T03:36:07.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.053 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.053 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.053 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.054 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.277974+0000", 2023-10-20T03:36:07.054 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.054 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.054 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.054 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.277974+0000", 2023-10-20T03:36:07.055 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.055 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.055 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.055 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.277977+0000", 2023-10-20T03:36:07.056 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.553e-06 2023-10-20T03:36:07.056 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.056 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.056 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.056 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278381+0000", 2023-10-20T03:36:07.057 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00040455200000000003 2023-10-20T03:36:07.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.057 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.057 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.057 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278416+0000", 2023-10-20T03:36:07.058 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4269e-05 2023-10-20T03:36:07.058 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.058 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.058 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.058 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278440+0000", 2023-10-20T03:36:07.059 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3926000000000001e-05 2023-10-20T03:36:07.059 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.059 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.059 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.059 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.287724+0000", 2023-10-20T03:36:07.059 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0092847510000000008 2023-10-20T03:36:07.060 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.060 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.060 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:07.060 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.288029+0000", 2023-10-20T03:36:07.060 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00030455899999999998 2023-10-20T03:36:07.061 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.061 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.061 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.061 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.289828+0000", 2023-10-20T03:36:07.061 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017987210000000001 2023-10-20T03:36:07.061 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.062 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.062 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.062 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.300205+0000", 2023-10-20T03:36:07.062 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010377752000000001 2023-10-20T03:36:07.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.063 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.063 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:07.063 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.300337+0000", 2023-10-20T03:36:07.063 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000131755 2023-10-20T03:36:07.063 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.064 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.064 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.064 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.301112+0000", 2023-10-20T03:36:07.064 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00077523499999999999 2023-10-20T03:36:07.064 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.064 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.065 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.065 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.065 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.065 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:210 3.es0 3:7dffa048:::smithi17039300-5:head [setxattr _header (55) in=7b] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:36:07.065 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.278490+0000", 2023-10-20T03:36:07.065 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 47.753957323999998, 2023-10-20T03:36:07.066 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027549357999999999, 2023-10-20T03:36:07.066 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.066 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:07.066 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:07.066 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:07.067 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:07.067 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210 2023-10-20T03:36:07.067 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.067 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.067 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.068 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.068 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278490+0000", 2023-10-20T03:36:07.068 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.068 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.068 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.068 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.069 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278490+0000", 2023-10-20T03:36:07.069 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.069 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.069 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.069 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.069 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278492+0000", 2023-10-20T03:36:07.070 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.59e-06 2023-10-20T03:36:07.070 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.070 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.070 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.070 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278501+0000", 2023-10-20T03:36:07.070 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.986e-06 2023-10-20T03:36:07.071 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.071 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.071 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.071 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278519+0000", 2023-10-20T03:36:07.071 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8238999999999999e-05 2023-10-20T03:36:07.072 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.072 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.072 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.072 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278536+0000", 2023-10-20T03:36:07.072 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7376000000000001e-05 2023-10-20T03:36:07.073 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.073 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.073 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.073 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.301212+0000", 2023-10-20T03:36:07.073 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.022675563999999999 2023-10-20T03:36:07.073 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.074 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.074 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:07.074 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.301368+0000", 2023-10-20T03:36:07.074 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015616 2023-10-20T03:36:07.074 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.075 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.075 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.075 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.302117+0000", 2023-10-20T03:36:07.075 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00074924300000000002 2023-10-20T03:36:07.075 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.075 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.076 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.076 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.303929+0000", 2023-10-20T03:36:07.076 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0018115830000000001 2023-10-20T03:36:07.076 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.076 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.077 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:07.077 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.305714+0000", 2023-10-20T03:36:07.077 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001785502 2023-10-20T03:36:07.077 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.077 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.077 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.078 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.306040+0000", 2023-10-20T03:36:07.078 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00032511500000000001 2023-10-20T03:36:07.078 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.078 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.078 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.079 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.079 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.079 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:211 3.es0 3:7dffa048:::smithi17039300-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:36:07.079 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.278563+0000", 2023-10-20T03:36:07.079 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 47.753885017000002, 2023-10-20T03:36:07.079 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030205164, 2023-10-20T03:36:07.080 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.080 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:07.080 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:07.080 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:07.080 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:07.081 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211 2023-10-20T03:36:07.081 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.081 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.081 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.081 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.081 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278563+0000", 2023-10-20T03:36:07.082 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.082 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.082 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.082 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.082 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278563+0000", 2023-10-20T03:36:07.083 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.083 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.083 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.083 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.083 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278564+0000", 2023-10-20T03:36:07.084 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.826e-06 2023-10-20T03:36:07.084 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.084 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.084 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.084 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278572+0000", 2023-10-20T03:36:07.084 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0299999999999994e-06 2023-10-20T03:36:07.085 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.085 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.085 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.085 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278598+0000", 2023-10-20T03:36:07.085 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5593e-05 2023-10-20T03:36:07.086 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.086 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.086 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.086 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278619+0000", 2023-10-20T03:36:07.086 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0755e-05 2023-10-20T03:36:07.087 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.087 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.087 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.087 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.302759+0000", 2023-10-20T03:36:07.087 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.024140663 2023-10-20T03:36:07.087 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.088 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.088 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:07.088 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.303005+0000", 2023-10-20T03:36:07.088 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024529499999999997 2023-10-20T03:36:07.088 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.089 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.089 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.089 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.306691+0000", 2023-10-20T03:36:07.089 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0036863809999999999 2023-10-20T03:36:07.089 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.089 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.090 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:07.090 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.306820+0000", 2023-10-20T03:36:07.090 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012892200000000001 2023-10-20T03:36:07.090 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.090 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.091 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.091 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.308768+0000", 2023-10-20T03:36:07.091 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001947699 2023-10-20T03:36:07.091 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.091 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.092 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.092 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.092 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.092 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.ds2 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:07.092 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.388826+0000", 2023-10-20T03:36:07.092 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 40.643621928000002, 2023-10-20T03:36:07.093 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027218437000000002, 2023-10-20T03:36:07.093 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.093 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.093 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.093 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.094 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.094 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388826+0000", 2023-10-20T03:36:07.094 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.094 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.094 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.095 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.095 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388826+0000", 2023-10-20T03:36:07.095 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.095 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.095 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.096 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.096 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388827+0000", 2023-10-20T03:36:07.096 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5300000000000002e-07 2023-10-20T03:36:07.096 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.096 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.097 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.097 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389257+0000", 2023-10-20T03:36:07.097 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00043021500000000001 2023-10-20T03:36:07.097 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.097 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.097 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.098 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389281+0000", 2023-10-20T03:36:07.098 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4697000000000001e-05 2023-10-20T03:36:07.098 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.098 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.098 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.099 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389295+0000", 2023-10-20T03:36:07.099 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3491e-05 2023-10-20T03:36:07.099 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.099 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.099 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.100 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389419+0000", 2023-10-20T03:36:07.100 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012411400000000001 2023-10-20T03:36:07.100 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.100 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.100 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.101 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389432+0000", 2023-10-20T03:36:07.101 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2551000000000001e-05 2023-10-20T03:36:07.101 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.101 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.101 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.101 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.415986+0000", 2023-10-20T03:36:07.102 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026554069999999999 2023-10-20T03:36:07.102 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.102 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.102 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.102 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.416044+0000", 2023-10-20T03:36:07.103 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8346e-05 2023-10-20T03:36:07.103 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.103 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.103 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.103 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.103 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.104 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.ds2 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:07.104 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.395618+0000", 2023-10-20T03:36:07.104 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 40.636829792, 2023-10-20T03:36:07.104 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.02052816, 2023-10-20T03:36:07.104 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.105 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.105 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.105 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.105 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.105 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395618+0000", 2023-10-20T03:36:07.105 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.106 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.106 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.106 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.106 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395618+0000", 2023-10-20T03:36:07.106 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.107 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.107 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.107 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.107 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395619+0000", 2023-10-20T03:36:07.107 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5400000000000001e-06 2023-10-20T03:36:07.107 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.108 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.108 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.108 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395915+0000", 2023-10-20T03:36:07.108 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00029547800000000001 2023-10-20T03:36:07.108 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.109 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.109 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.109 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395953+0000", 2023-10-20T03:36:07.109 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8164000000000003e-05 2023-10-20T03:36:07.109 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.110 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.110 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.110 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.395971+0000", 2023-10-20T03:36:07.110 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8484000000000001e-05 2023-10-20T03:36:07.110 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.110 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.111 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.111 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.399211+0000", 2023-10-20T03:36:07.111 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0032393040000000001 2023-10-20T03:36:07.111 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.111 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.112 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.112 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.399239+0000", 2023-10-20T03:36:07.112 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7991999999999999e-05 2023-10-20T03:36:07.112 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.112 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.113 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.113 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.416107+0000", 2023-10-20T03:36:07.113 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016868095 2023-10-20T03:36:07.113 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.113 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.113 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.114 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.416146+0000", 2023-10-20T03:36:07.114 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9103000000000002e-05 2023-10-20T03:36:07.114 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.114 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.114 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.115 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.115 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.115 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.ds2 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:36:07.115 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:30.392840+0000", 2023-10-20T03:36:07.115 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 36.639607722000001, 2023-10-20T03:36:07.115 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.037253397000000001, 2023-10-20T03:36:07.116 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.116 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.116 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.116 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.116 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.117 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.392840+0000", 2023-10-20T03:36:07.117 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.117 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.117 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.117 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.118 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.392840+0000", 2023-10-20T03:36:07.118 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.118 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.118 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.118 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.118 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.392841+0000", 2023-10-20T03:36:07.119 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.567e-06 2023-10-20T03:36:07.119 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.119 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.119 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.119 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.394858+0000", 2023-10-20T03:36:07.120 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020165589999999998 2023-10-20T03:36:07.120 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.120 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.120 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.120 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.394892+0000", 2023-10-20T03:36:07.120 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4001999999999997e-05 2023-10-20T03:36:07.121 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.121 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.121 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.121 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.394916+0000", 2023-10-20T03:36:07.121 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4002000000000002e-05 2023-10-20T03:36:07.122 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.122 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.122 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.122 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.395085+0000", 2023-10-20T03:36:07.122 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016879899999999999 2023-10-20T03:36:07.123 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.123 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.123 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.123 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.395112+0000", 2023-10-20T03:36:07.123 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7032000000000001e-05 2023-10-20T03:36:07.123 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.124 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.124 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.124 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.430024+0000", 2023-10-20T03:36:07.124 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.034911958 2023-10-20T03:36:07.124 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.125 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.125 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.125 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.430093+0000", 2023-10-20T03:36:07.125 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9478000000000001e-05 2023-10-20T03:36:07.125 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.125 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.126 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.126 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.126 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.126 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs5 53/47 ECSubWrite(tid=56, reqid=osd.3.0:55, at_version=0'0, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:07.126 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.452318+0000", 2023-10-20T03:36:07.127 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.580129120999999, 2023-10-20T03:36:07.127 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019749735000000001, 2023-10-20T03:36:07.127 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.127 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.127 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.128 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.128 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.128 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452318+0000", 2023-10-20T03:36:07.128 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.128 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.128 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.129 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.129 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452318+0000", 2023-10-20T03:36:07.129 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.129 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.129 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.129 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.130 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452321+0000", 2023-10-20T03:36:07.130 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2670000000000001e-06 2023-10-20T03:36:07.130 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.130 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.130 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.131 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452350+0000", 2023-10-20T03:36:07.131 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8915999999999999e-05 2023-10-20T03:36:07.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.131 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.131 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.131 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452391+0000", 2023-10-20T03:36:07.132 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0992000000000003e-05 2023-10-20T03:36:07.132 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.132 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.132 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.132 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452414+0000", 2023-10-20T03:36:07.132 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3632999999999999e-05 2023-10-20T03:36:07.133 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.133 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.133 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.133 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452600+0000", 2023-10-20T03:36:07.133 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018559199999999999 2023-10-20T03:36:07.134 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.134 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.134 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.134 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452632+0000", 2023-10-20T03:36:07.134 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2298999999999997e-05 2023-10-20T03:36:07.135 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.135 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.135 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.135 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.471913+0000", 2023-10-20T03:36:07.135 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019280950000000002 2023-10-20T03:36:07.135 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.136 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.136 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.136 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.472068+0000", 2023-10-20T03:36:07.136 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000155086 2023-10-20T03:36:07.136 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.136 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.137 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.137 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.137 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.137 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs5 53/47 ECSubWrite(tid=57, reqid=client.4570.0:408, at_version=53'31, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:07.137 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.452469+0000", 2023-10-20T03:36:07.138 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.579978171, 2023-10-20T03:36:07.138 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.033778967999999999, 2023-10-20T03:36:07.138 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.138 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.138 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.139 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.139 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.139 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452469+0000", 2023-10-20T03:36:07.139 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.139 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.140 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.140 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.140 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452469+0000", 2023-10-20T03:36:07.140 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.140 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.140 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.141 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.141 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452470+0000", 2023-10-20T03:36:07.141 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6299999999999993e-07 2023-10-20T03:36:07.141 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.141 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.142 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.142 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452906+0000", 2023-10-20T03:36:07.142 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00043586900000000002 2023-10-20T03:36:07.142 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.142 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.143 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.143 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452933+0000", 2023-10-20T03:36:07.143 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.69e-05 2023-10-20T03:36:07.143 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.143 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.143 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.144 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452952+0000", 2023-10-20T03:36:07.144 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9089e-05 2023-10-20T03:36:07.144 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.144 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.144 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.145 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.453188+0000", 2023-10-20T03:36:07.145 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00023577799999999999 2023-10-20T03:36:07.145 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.145 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.145 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.145 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.453218+0000", 2023-10-20T03:36:07.146 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0442000000000001e-05 2023-10-20T03:36:07.146 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.146 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.146 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.146 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.486138+0000", 2023-10-20T03:36:07.147 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.032919118999999997 2023-10-20T03:36:07.147 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.147 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.147 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.147 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.486248+0000", 2023-10-20T03:36:07.148 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110808 2023-10-20T03:36:07.148 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.148 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.148 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.148 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.148 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.149 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:432 3.es0 3:7b3c804c:::smithi17039300-40:head [create,setxattr [VYQXaeer[Nsq`eTbgjNZfdhqbTRn_][lloSeZUP (16840) in=40b,setxattr hpaXc^]qjrqU_ciY[cWncm]SPa^tP_jRrRXm`jol (175) in=40b,setxattr YVpShpT[fhfamkUsgr]mb\\foasmX^k`jdhbbdlfP (1823) in=40b,setxattr ZhfkQl`OllQo^hemUnadbmqWdoabfgjcc^sVm_gf (1611) in=40b,setxattr aakOOaWemYRPYcaZ\\knQb (1051) in=21b,setxattr gPaagTUadWkOtqaRt]^rdUj^Zinmt]U^ (1051) in=32b,setxattr SfXmSZ]crX[dqcaqjNrrroQQbptW^Q^nUSUlcgdk (807) in=40b,setxattr SmRaehVPpUNpfZ[UX\\ZO_Ygfofk^ah^Xki^XTeX` (1191) in=40b,setxattr aTgW^YtXrjnrhT (1051) in=14b,setxattr idO[Vrlsc^pW[hYfh`UQSs^m_mU[YefWeld[YZ[c (1543) in=40b,setxattr ]Q``task\\rr]qaY[Pg`i[alh[rPlSdrcQ^iW^Z^g (1157) in=40b,setxattr _OOWpoQQg\\jtQUi``q_s[ae_akq`YOYilNTq^SPO (1121) in=40b,setxattr kN\\_ZaOX]q[kXen^Z]liYfaVP]prk\\r\\\\_Wsd[d_ (105) in=40b,setxattr OU`jdr^jTir[joOegmm (1051) in=19b,setxattr tRofbjROgtWqoRfmUVZXhpaZ[Q^sUNZb`\\YsdUfh (1507) in=40b,setxattr RRNpbt[`okmbR]RpSWfmmedXrUpaltcbWPbNRZRZ (67) in=40b,setxattr i[`^r^aojXRaU[[T\\kgWN^ (1051) in=22b,setxattr gZhSajUgdm`o]W[X`sgdVl^TtY^_SX[_l[Tr_aaO (663) in=40b,setxattr qN][QajljeddtQTcQX_QaTrg\\V\\pY\\QbsWehde`s (1646) in=40b,setxattr fbTrNt\\`ZVPVN[]tcg]iPXYOnmYnV]oaW_^o^gdr (1403) in=40b] snapc 16=[16,3,2] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2023-10-20T03:36:07.149 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:37.439657+0000", 2023-10-20T03:36:07.149 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 29.592790952000001, 2023-10-20T03:36:07.149 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.021990928, 2023-10-20T03:36:07.149 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.150 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:07.150 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:07.150 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:07.150 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:07.150 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 432 2023-10-20T03:36:07.151 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.151 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.151 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.151 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.151 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439657+0000", 2023-10-20T03:36:07.151 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.152 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.152 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.152 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.152 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439657+0000", 2023-10-20T03:36:07.152 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.153 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.153 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.153 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.153 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439660+0000", 2023-10-20T03:36:07.153 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.433e-06 2023-10-20T03:36:07.153 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.154 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.154 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.154 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439750+0000", 2023-10-20T03:36:07.154 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9494999999999996e-05 2023-10-20T03:36:07.154 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.155 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.155 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.155 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439790+0000", 2023-10-20T03:36:07.155 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0914999999999997e-05 2023-10-20T03:36:07.155 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.155 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.156 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.156 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439815+0000", 2023-10-20T03:36:07.156 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4586000000000001e-05 2023-10-20T03:36:07.156 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.156 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.157 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.157 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.439997+0000", 2023-10-20T03:36:07.157 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000181893 2023-10-20T03:36:07.157 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.157 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.157 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op must wait for map", 2023-10-20T03:36:07.158 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.440020+0000", 2023-10-20T03:36:07.158 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2685999999999999e-05 2023-10-20T03:36:07.158 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.158 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.158 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.158 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.445780+0000", 2023-10-20T03:36:07.159 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.005760818 2023-10-20T03:36:07.159 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.159 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.159 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:07.159 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.445992+0000", 2023-10-20T03:36:07.160 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00021181699999999999 2023-10-20T03:36:07.160 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.160 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.160 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.160 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.447439+0000", 2023-10-20T03:36:07.160 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0014467530000000001 2023-10-20T03:36:07.161 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.161 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.161 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.161 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.451095+0000", 2023-10-20T03:36:07.161 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0036556359999999999 2023-10-20T03:36:07.161 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.162 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.162 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:07.162 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.461212+0000", 2023-10-20T03:36:07.162 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010117477 2023-10-20T03:36:07.162 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.163 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.163 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.163 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.461648+0000", 2023-10-20T03:36:07.163 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00043541899999999999 2023-10-20T03:36:07.163 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.163 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.164 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.164 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.164 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.164 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs5 55/47 ECSubWrite(tid=71, reqid=client.4570.0:406, at_version=55'38, trim_to=0'0, roll_forward_to=54'36))", 2023-10-20T03:36:07.164 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:37.472798+0000", 2023-10-20T03:36:07.164 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 29.559649402000002, 2023-10-20T03:36:07.165 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019968949999999999, 2023-10-20T03:36:07.165 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.165 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.165 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.165 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.166 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.166 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.472798+0000", 2023-10-20T03:36:07.166 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.166 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.166 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.166 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.167 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.472798+0000", 2023-10-20T03:36:07.167 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.167 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.167 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.167 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.167 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.472800+0000", 2023-10-20T03:36:07.168 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.654e-06 2023-10-20T03:36:07.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.168 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.168 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.168 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473206+0000", 2023-10-20T03:36:07.168 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00040649700000000002 2023-10-20T03:36:07.169 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.169 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.169 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.169 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473249+0000", 2023-10-20T03:36:07.169 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2691999999999997e-05 2023-10-20T03:36:07.169 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.170 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.170 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.170 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473268+0000", 2023-10-20T03:36:07.170 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9355999999999999e-05 2023-10-20T03:36:07.170 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.170 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.171 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.171 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473402+0000", 2023-10-20T03:36:07.171 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013405700000000001 2023-10-20T03:36:07.171 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.171 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.171 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.172 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473418+0000", 2023-10-20T03:36:07.172 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6028999999999998e-05 2023-10-20T03:36:07.172 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.172 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.172 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.172 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.492673+0000", 2023-10-20T03:36:07.173 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019254454000000001 2023-10-20T03:36:07.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.173 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.173 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.173 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.492767+0000", 2023-10-20T03:36:07.173 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4210999999999996e-05 2023-10-20T03:36:07.174 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.174 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.174 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.174 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.174 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.174 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.7s1 60/31 ECSubWrite(tid=198, reqid=client.4570.0:458, at_version=60'8, trim_to=0'0, roll_forward_to=38'7))", 2023-10-20T03:36:07.175 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.888718+0000", 2023-10-20T03:36:07.175 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 25.143729823000001, 2023-10-20T03:36:07.175 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027011496999999999, 2023-10-20T03:36:07.175 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.175 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.175 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.176 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.176 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.176 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888718+0000", 2023-10-20T03:36:07.176 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.176 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.177 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.177 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.177 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888718+0000", 2023-10-20T03:36:07.177 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.177 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.177 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.178 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.178 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888719+0000", 2023-10-20T03:36:07.178 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7230000000000001e-06 2023-10-20T03:36:07.178 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.178 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.178 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.179 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889432+0000", 2023-10-20T03:36:07.179 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00071276 2023-10-20T03:36:07.179 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.179 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.179 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.179 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889488+0000", 2023-10-20T03:36:07.180 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5309000000000002e-05 2023-10-20T03:36:07.180 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.180 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.180 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.180 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889511+0000", 2023-10-20T03:36:07.180 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3572999999999999e-05 2023-10-20T03:36:07.181 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.181 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.181 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.181 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889746+0000", 2023-10-20T03:36:07.181 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00023487099999999999 2023-10-20T03:36:07.181 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.182 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.182 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.182 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889773+0000", 2023-10-20T03:36:07.182 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7330000000000001e-05 2023-10-20T03:36:07.182 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.182 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.183 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.183 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.915601+0000", 2023-10-20T03:36:07.183 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.02582746 2023-10-20T03:36:07.183 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.183 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.183 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.184 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.915729+0000", 2023-10-20T03:36:07.184 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012847100000000001 2023-10-20T03:36:07.184 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.184 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.184 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.184 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.185 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.185 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.7s1 60/31 ECSubWrite(tid=200, reqid=client.4570.0:459, at_version=60'9, trim_to=0'0, roll_forward_to=38'7))", 2023-10-20T03:36:07.185 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.898136+0000", 2023-10-20T03:36:07.185 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 25.134311288999999, 2023-10-20T03:36:07.185 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019686072999999998, 2023-10-20T03:36:07.186 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.186 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.186 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.186 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.186 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.186 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.898136+0000", 2023-10-20T03:36:07.187 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.187 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.187 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.187 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.187 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.898136+0000", 2023-10-20T03:36:07.187 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.188 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.188 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.188 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.898138+0000", 2023-10-20T03:36:07.188 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.271e-06 2023-10-20T03:36:07.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.189 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.189 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.189 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.898567+0000", 2023-10-20T03:36:07.189 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00042924700000000001 2023-10-20T03:36:07.189 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.190 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.190 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.190 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.898584+0000", 2023-10-20T03:36:07.190 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7260000000000001e-05 2023-10-20T03:36:07.190 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.190 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.191 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.191 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.898622+0000", 2023-10-20T03:36:07.191 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8300999999999998e-05 2023-10-20T03:36:07.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.191 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.191 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.192 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.898733+0000", 2023-10-20T03:36:07.192 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110472 2023-10-20T03:36:07.192 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.192 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.192 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.193 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.898750+0000", 2023-10-20T03:36:07.193 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6988000000000001e-05 2023-10-20T03:36:07.193 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.193 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.193 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.193 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.917774+0000", 2023-10-20T03:36:07.194 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019023827 2023-10-20T03:36:07.194 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.194 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.194 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.194 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.917822+0000", 2023-10-20T03:36:07.195 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8707e-05 2023-10-20T03:36:07.195 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.195 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.195 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.195 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.195 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.196 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.bs4 70/56 ECSubWrite(tid=261, reqid=client.4570.0:562, at_version=70'37, trim_to=0'0, roll_forward_to=61'34))", 2023-10-20T03:36:07.196 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:52.038664+0000", 2023-10-20T03:36:07.196 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.993783351999999, 2023-10-20T03:36:07.196 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025138243000000001, 2023-10-20T03:36:07.196 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.197 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.197 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.197 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.197 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.197 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.038664+0000", 2023-10-20T03:36:07.198 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.198 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.198 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.198 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.198 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.038664+0000", 2023-10-20T03:36:07.198 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.199 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.199 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.199 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.199 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.038665+0000", 2023-10-20T03:36:07.199 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4e-07 2023-10-20T03:36:07.200 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.200 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.200 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.200 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.038993+0000", 2023-10-20T03:36:07.200 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00032772600000000003 2023-10-20T03:36:07.200 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.201 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.201 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.201 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.039013+0000", 2023-10-20T03:36:07.201 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0412e-05 2023-10-20T03:36:07.201 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.202 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.202 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.202 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.039024+0000", 2023-10-20T03:36:07.202 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0988000000000001e-05 2023-10-20T03:36:07.202 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.203 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.203 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.203 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.039177+0000", 2023-10-20T03:36:07.203 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000152831 2023-10-20T03:36:07.203 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.203 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.204 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.204 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.039230+0000", 2023-10-20T03:36:07.204 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2528999999999998e-05 2023-10-20T03:36:07.204 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.204 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.204 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.205 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.063687+0000", 2023-10-20T03:36:07.205 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.024457656000000001 2023-10-20T03:36:07.205 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.205 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.205 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.206 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.063802+0000", 2023-10-20T03:36:07.206 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000115161 2023-10-20T03:36:07.206 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.206 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.206 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.207 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.207 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:589 3.es0 3:7b3c804c:::smithi17039300-40:head [append 0~720896] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:07.207 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.001880+0000", 2023-10-20T03:36:07.207 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.030567997, 2023-10-20T03:36:07.207 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.047597967999999997, 2023-10-20T03:36:07.208 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.208 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:07.208 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:07.208 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:07.208 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:07.208 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 589 2023-10-20T03:36:07.209 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.209 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.209 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.209 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.209 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.001880+0000", 2023-10-20T03:36:07.210 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.210 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.210 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.210 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.210 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.001880+0000", 2023-10-20T03:36:07.211 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.211 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.211 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.211 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.211 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.001882+0000", 2023-10-20T03:36:07.211 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2139999999999999e-06 2023-10-20T03:36:07.212 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.212 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.212 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.212 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.002268+0000", 2023-10-20T03:36:07.212 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00038638700000000003 2023-10-20T03:36:07.212 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.213 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.213 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.213 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.002297+0000", 2023-10-20T03:36:07.213 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8697e-05 2023-10-20T03:36:07.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.214 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.214 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.214 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.002348+0000", 2023-10-20T03:36:07.214 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1187999999999998e-05 2023-10-20T03:36:07.214 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.214 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.215 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.215 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.002561+0000", 2023-10-20T03:36:07.215 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000213427 2023-10-20T03:36:07.215 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.215 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.215 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:07.216 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.002733+0000", 2023-10-20T03:36:07.216 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000171256 2023-10-20T03:36:07.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.216 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.216 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.217 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004158+0000", 2023-10-20T03:36:07.217 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0014255330000000001 2023-10-20T03:36:07.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.217 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.217 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.039236+0000", 2023-10-20T03:36:07.218 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.035078249999999998 2023-10-20T03:36:07.218 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.218 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:07.218 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.049317+0000", 2023-10-20T03:36:07.218 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010080458 2023-10-20T03:36:07.219 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.219 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.219 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.219 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.049478+0000", 2023-10-20T03:36:07.219 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016055799999999999 2023-10-20T03:36:07.219 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.220 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.220 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.220 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.220 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.220 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:590 3.es0 3:7b3c804c:::smithi17039300-40:head [append 0~802816] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:07.221 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.009044+0000", 2023-10-20T03:36:07.221 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.023403267000001, 2023-10-20T03:36:07.221 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.041318410999999999, 2023-10-20T03:36:07.221 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.221 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:07.221 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:07.222 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:07.222 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:07.222 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 590 2023-10-20T03:36:07.222 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.222 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.222 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.223 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.223 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.009044+0000", 2023-10-20T03:36:07.223 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.223 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.223 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.223 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.224 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.009044+0000", 2023-10-20T03:36:07.224 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.224 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.224 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.009047+0000", 2023-10-20T03:36:07.225 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6879999999999999e-06 2023-10-20T03:36:07.225 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.225 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.225 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.225 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.009793+0000", 2023-10-20T03:36:07.225 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00074570099999999998 2023-10-20T03:36:07.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.226 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.226 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.226 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.009828+0000", 2023-10-20T03:36:07.226 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5045000000000001e-05 2023-10-20T03:36:07.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.227 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.227 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.227 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.009851+0000", 2023-10-20T03:36:07.227 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3206000000000001e-05 2023-10-20T03:36:07.227 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.227 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.228 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.228 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.011468+0000", 2023-10-20T03:36:07.228 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001617402 2023-10-20T03:36:07.228 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.228 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.228 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:07.229 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.011751+0000", 2023-10-20T03:36:07.229 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000283107 2023-10-20T03:36:07.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.229 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.229 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.229 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.014038+0000", 2023-10-20T03:36:07.230 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002286695 2023-10-20T03:36:07.230 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.230 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.230 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.230 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.039572+0000", 2023-10-20T03:36:07.231 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025533541999999999 2023-10-20T03:36:07.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.231 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.231 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:07.231 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.049801+0000", 2023-10-20T03:36:07.231 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010229548999999999 2023-10-20T03:36:07.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.232 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.232 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.232 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.050363+0000", 2023-10-20T03:36:07.232 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00056147599999999999 2023-10-20T03:36:07.232 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.233 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.233 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.233 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.233 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.233 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:591 3.es0 3:7b3c804c:::smithi17039300-40:head [append 0~360448] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:07.233 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.012685+0000", 2023-10-20T03:36:07.234 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.019762988, 2023-10-20T03:36:07.234 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.038593720999999998, 2023-10-20T03:36:07.234 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.234 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:07.234 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:07.234 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:07.235 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:07.235 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 591 2023-10-20T03:36:07.235 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.235 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.235 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.235 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.236 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012685+0000", 2023-10-20T03:36:07.236 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.236 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.236 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.236 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.236 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012685+0000", 2023-10-20T03:36:07.237 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.237 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.237 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.237 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012686+0000", 2023-10-20T03:36:07.237 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.429e-06 2023-10-20T03:36:07.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.238 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.238 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.238 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012926+0000", 2023-10-20T03:36:07.238 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024043099999999999 2023-10-20T03:36:07.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.239 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.239 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.239 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012940+0000", 2023-10-20T03:36:07.239 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3447e-05 2023-10-20T03:36:07.239 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.239 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.240 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.240 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012947+0000", 2023-10-20T03:36:07.240 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4590000000000001e-06 2023-10-20T03:36:07.240 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.240 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.240 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.241 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.034646+0000", 2023-10-20T03:36:07.241 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.021698205000000002 2023-10-20T03:36:07.241 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.241 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.241 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:07.241 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.034733+0000", 2023-10-20T03:36:07.242 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7528000000000006e-05 2023-10-20T03:36:07.242 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.242 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.242 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.242 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.035416+0000", 2023-10-20T03:36:07.242 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00068264400000000005 2023-10-20T03:36:07.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.243 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.243 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.243 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.042809+0000", 2023-10-20T03:36:07.243 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0073934220000000002 2023-10-20T03:36:07.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.244 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.244 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:07.244 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.050705+0000", 2023-10-20T03:36:07.244 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0078962830000000005 2023-10-20T03:36:07.244 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.244 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.245 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.245 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.051278+0000", 2023-10-20T03:36:07.245 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000572873 2023-10-20T03:36:07.245 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.245 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.245 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.246 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.246 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.246 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:592 3.es0 3:7b3c804c:::smithi17039300-40:head [setxattr _header (57) in=7b] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:07.246 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.012969+0000", 2023-10-20T03:36:07.246 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.019478169999999, 2023-10-20T03:36:07.247 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039254223999999997, 2023-10-20T03:36:07.247 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.247 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:07.247 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:07.247 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:07.248 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:07.248 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 592 2023-10-20T03:36:07.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.248 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.248 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.248 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.249 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012969+0000", 2023-10-20T03:36:07.249 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.249 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.249 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.249 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.249 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012969+0000", 2023-10-20T03:36:07.250 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.250 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.250 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012970+0000", 2023-10-20T03:36:07.251 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4799999999999997e-07 2023-10-20T03:36:07.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.251 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.251 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.251 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012975+0000", 2023-10-20T03:36:07.252 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6160000000000004e-06 2023-10-20T03:36:07.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.252 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.252 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012984+0000", 2023-10-20T03:36:07.252 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0850000000000003e-06 2023-10-20T03:36:07.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.253 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.253 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.253 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012992+0000", 2023-10-20T03:36:07.253 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6270000000000006e-06 2023-10-20T03:36:07.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.254 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.254 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.039656+0000", 2023-10-20T03:36:07.254 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.02666342 2023-10-20T03:36:07.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.255 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:07.255 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.039948+0000", 2023-10-20T03:36:07.255 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000292458 2023-10-20T03:36:07.255 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.255 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.041465+0000", 2023-10-20T03:36:07.256 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00151659 2023-10-20T03:36:07.256 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.256 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.043158+0000", 2023-10-20T03:36:07.257 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0016926210000000001 2023-10-20T03:36:07.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.257 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:07.257 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.051619+0000", 2023-10-20T03:36:07.257 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0084616410000000006 2023-10-20T03:36:07.258 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.258 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.258 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.258 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.052224+0000", 2023-10-20T03:36:07.258 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00060441800000000001 2023-10-20T03:36:07.259 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.259 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.259 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.259 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.259 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.259 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:593 3.es0 3:7b3c804c:::smithi17039300-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:07.260 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.013003+0000", 2023-10-20T03:36:07.260 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.019444893999999, 2023-10-20T03:36:07.260 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.041417311999999998, 2023-10-20T03:36:07.260 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.260 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:07.260 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:07.261 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:07.261 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:07.261 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 593 2023-10-20T03:36:07.261 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.261 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.261 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.262 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.262 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013003+0000", 2023-10-20T03:36:07.262 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.263 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013003+0000", 2023-10-20T03:36:07.263 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.264 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013003+0000", 2023-10-20T03:36:07.264 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8299999999999997e-07 2023-10-20T03:36:07.264 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.264 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.264 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.264 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013006+0000", 2023-10-20T03:36:07.265 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6699999999999998e-06 2023-10-20T03:36:07.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.265 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.265 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013015+0000", 2023-10-20T03:36:07.266 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3479999999999993e-06 2023-10-20T03:36:07.266 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.266 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.266 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.266 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013021+0000", 2023-10-20T03:36:07.267 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7880000000000003e-06 2023-10-20T03:36:07.267 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.267 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.267 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.267 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.042224+0000", 2023-10-20T03:36:07.267 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029202676 2023-10-20T03:36:07.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.268 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.268 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:07.268 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.042562+0000", 2023-10-20T03:36:07.268 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000338029 2023-10-20T03:36:07.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.269 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.269 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.053183+0000", 2023-10-20T03:36:07.269 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010621619000000001 2023-10-20T03:36:07.269 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.270 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:07.270 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.053311+0000", 2023-10-20T03:36:07.270 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012811800000000001 2023-10-20T03:36:07.270 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.270 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.271 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.271 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.054420+0000", 2023-10-20T03:36:07.271 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0011084809999999999 2023-10-20T03:36:07.271 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.271 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.271 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.272 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.272 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWriteReply(3.es0 71/31 ECSubWriteReply(tid=52, last_complete=71'24, committed=1, applied=1))", 2023-10-20T03:36:07.272 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.013893+0000", 2023-10-20T03:36:07.272 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.018554376000001, 2023-10-20T03:36:07.273 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025527006000000001, 2023-10-20T03:36:07.273 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.273 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.273 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.273 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.273 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.274 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013893+0000", 2023-10-20T03:36:07.274 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.274 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.274 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013893+0000", 2023-10-20T03:36:07.275 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.275 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.275 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.275 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.275 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013894+0000", 2023-10-20T03:36:07.276 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.8899999999999998e-07 2023-10-20T03:36:07.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.276 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.276 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013900+0000", 2023-10-20T03:36:07.276 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8420000000000001e-06 2023-10-20T03:36:07.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.277 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.277 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013912+0000", 2023-10-20T03:36:07.277 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1626e-05 2023-10-20T03:36:07.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.278 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.278 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013919+0000", 2023-10-20T03:36:07.278 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7500000000000003e-06 2023-10-20T03:36:07.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.279 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.279 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.039351+0000", 2023-10-20T03:36:07.279 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025431287 2023-10-20T03:36:07.279 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.279 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.279 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.280 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.039420+0000", 2023-10-20T03:36:07.280 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9611999999999997e-05 2023-10-20T03:36:07.280 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.280 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.280 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.280 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.281 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.281 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:07.282 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-10-20T03:36:07.284 INFO:tasks.rados.rados.0.smithi170.stdout:285: done (1 left) 2023-10-20T03:36:07.284 INFO:tasks.rados.rados.0.smithi170.stdout:288: done (0 left) 2023-10-20T03:36:07.284 INFO:tasks.rados.rados.0.smithi170.stdout:289: read oid 50 snap -1 2023-10-20T03:36:07.284 INFO:tasks.rados.rados.0.smithi170.stdout:289: expect (ObjNum 122 snap 25 seq_num 122) 2023-10-20T03:36:07.284 INFO:tasks.rados.rados.0.smithi170.stdout:290: rollback oid 37 current snap is 26 2023-10-20T03:36:07.285 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 37 to 24 2023-10-20T03:36:07.285 INFO:tasks.rados.rados.0.smithi170.stdout:291: delete oid 27 current snap is 26 2023-10-20T03:36:07.285 INFO:tasks.rados.rados.0.smithi170.stdout:291: done (2 left) 2023-10-20T03:36:07.285 INFO:tasks.rados.rados.0.smithi170.stdout:292: delete oid 26 current snap is 26 2023-10-20T03:36:07.285 INFO:tasks.rados.rados.0.smithi170.stdout:290: finishing rollback tid 0 to smithi17039300-37 2023-10-20T03:36:07.286 INFO:tasks.rados.rados.0.smithi170.stdout:290: finishing rollback tid 1 to smithi17039300-37 2023-10-20T03:36:07.286 INFO:tasks.rados.rados.0.smithi170.stdout:290: finishing rollback tid 2 to smithi17039300-37 2023-10-20T03:36:07.286 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 37 v 64 (ObjNum 85 snap 9 seq_num 85) dirty exists 2023-10-20T03:36:07.286 INFO:tasks.rados.rados.0.smithi170.stdout:290: done (2 left) 2023-10-20T03:36:07.286 INFO:tasks.rados.rados.0.smithi170.stdout:292: done (1 left) 2023-10-20T03:36:07.287 INFO:tasks.rados.rados.0.smithi170.stdout:293: rmattr oid 32 current snap is 26 2023-10-20T03:36:07.287 INFO:tasks.rados.rados.0.smithi170.stdout:293: done (1 left) 2023-10-20T03:36:07.287 INFO:tasks.rados.rados.0.smithi170.stdout:294: snap_remove snap 24 2023-10-20T03:36:07.287 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:36:07.280+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:36:07.384 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:36:07.380+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:36:07.417 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:07.418 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:36:07.418 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2023-10-20T03:36:07.418 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:07.431 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-10-20T03:36:07.484 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:07.481+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:07.553 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:07.553 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:36:07.553 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:36:07.553 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:36:07.554 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:07.567 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-10-20T03:36:07.586 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:07.580+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:07.686 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:07.683+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:07.729 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:07.730 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2023-10-20T03:36:07.730 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2023-10-20T03:36:07.730 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2023-10-20T03:36:07.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.730 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.6s6 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:36:07.731 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:25.357590+0000", 2023-10-20T03:36:07.731 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 42.370790153000002, 2023-10-20T03:36:07.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015294129, 2023-10-20T03:36:07.731 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.731 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.732 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.732 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.732 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.357590+0000", 2023-10-20T03:36:07.732 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.733 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.733 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.733 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.357590+0000", 2023-10-20T03:36:07.733 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.734 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.734 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.357590+0000", 2023-10-20T03:36:07.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5400000000000002e-07 2023-10-20T03:36:07.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.735 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.735 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.358304+0000", 2023-10-20T03:36:07.735 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00071352399999999999 2023-10-20T03:36:07.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.736 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.736 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.358341+0000", 2023-10-20T03:36:07.736 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7917999999999998e-05 2023-10-20T03:36:07.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.736 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.737 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.737 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.358367+0000", 2023-10-20T03:36:07.737 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5043000000000001e-05 2023-10-20T03:36:07.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.737 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.737 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.738 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.358584+0000", 2023-10-20T03:36:07.738 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000217915 2023-10-20T03:36:07.738 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.738 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.738 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.739 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.358610+0000", 2023-10-20T03:36:07.739 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5284999999999999e-05 2023-10-20T03:36:07.739 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.739 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.739 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.739 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.372826+0000", 2023-10-20T03:36:07.740 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014215768 2023-10-20T03:36:07.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.740 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.740 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.740 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.372884+0000", 2023-10-20T03:36:07.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8322000000000002e-05 2023-10-20T03:36:07.741 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.741 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.741 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.741 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.742 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 44/31 ECSubWrite(tid=18, reqid=client.4570.0:323, at_version=44'9, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:36:07.742 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.361265+0000", 2023-10-20T03:36:07.742 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 41.367115298000002, 2023-10-20T03:36:07.742 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016040175, 2023-10-20T03:36:07.742 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.743 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.743 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.743 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.743 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.743 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361265+0000", 2023-10-20T03:36:07.743 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.744 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.744 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.744 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361265+0000", 2023-10-20T03:36:07.744 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.745 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.745 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361265+0000", 2023-10-20T03:36:07.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9999999999999998e-07 2023-10-20T03:36:07.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.746 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.746 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361583+0000", 2023-10-20T03:36:07.746 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00031771700000000002 2023-10-20T03:36:07.746 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.747 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.747 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361616+0000", 2023-10-20T03:36:07.747 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3555e-05 2023-10-20T03:36:07.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.747 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.747 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.748 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361639+0000", 2023-10-20T03:36:07.748 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2238999999999998e-05 2023-10-20T03:36:07.748 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.748 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.748 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.748 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361965+0000", 2023-10-20T03:36:07.749 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00032612299999999999 2023-10-20T03:36:07.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.749 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.749 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.749 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361993+0000", 2023-10-20T03:36:07.750 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8535999999999999e-05 2023-10-20T03:36:07.750 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.750 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.750 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.750 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.377210+0000", 2023-10-20T03:36:07.751 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015216644 2023-10-20T03:36:07.751 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.751 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.751 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.751 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.377305+0000", 2023-10-20T03:36:07.751 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4860999999999999e-05 2023-10-20T03:36:07.752 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.752 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.752 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.753 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 44/31 ECSubWrite(tid=19, reqid=client.4570.0:324, at_version=44'10, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:36:07.753 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.381682+0000", 2023-10-20T03:36:07.753 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 41.346697468999999, 2023-10-20T03:36:07.753 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013179592, 2023-10-20T03:36:07.753 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.754 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.754 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.754 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.754 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.754 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.381682+0000", 2023-10-20T03:36:07.755 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.755 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.755 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.755 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.755 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.381682+0000", 2023-10-20T03:36:07.756 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.756 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.756 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.756 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.756 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.381684+0000", 2023-10-20T03:36:07.756 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.897e-06 2023-10-20T03:36:07.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.757 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.757 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.757 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.382005+0000", 2023-10-20T03:36:07.757 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000320649 2023-10-20T03:36:07.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.758 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.758 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.382041+0000", 2023-10-20T03:36:07.758 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6242000000000003e-05 2023-10-20T03:36:07.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.759 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.759 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.759 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.382067+0000", 2023-10-20T03:36:07.759 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6208999999999999e-05 2023-10-20T03:36:07.759 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.759 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.760 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.760 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.382198+0000", 2023-10-20T03:36:07.760 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013078300000000001 2023-10-20T03:36:07.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.760 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.761 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.761 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.382221+0000", 2023-10-20T03:36:07.761 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2858999999999999e-05 2023-10-20T03:36:07.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.761 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.761 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.762 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.394748+0000", 2023-10-20T03:36:07.762 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012526822999999999 2023-10-20T03:36:07.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.762 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.762 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.763 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.394862+0000", 2023-10-20T03:36:07.763 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011413000000000001 2023-10-20T03:36:07.763 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.763 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.763 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.763 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.764 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.764 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 49/31 ECSubWrite(tid=39, reqid=client.4570.0:390, at_version=49'23, trim_to=0'0, roll_forward_to=47'21))", 2023-10-20T03:36:07.764 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:31.415158+0000", 2023-10-20T03:36:07.764 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 36.313221788999996, 2023-10-20T03:36:07.764 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.020096919000000001, 2023-10-20T03:36:07.765 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.765 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.765 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.765 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.765 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.765 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415158+0000", 2023-10-20T03:36:07.766 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.766 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.766 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.766 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.766 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415158+0000", 2023-10-20T03:36:07.767 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.767 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.767 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.767 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.767 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415161+0000", 2023-10-20T03:36:07.768 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6690000000000002e-06 2023-10-20T03:36:07.768 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.768 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.768 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.768 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415623+0000", 2023-10-20T03:36:07.769 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00046182100000000001 2023-10-20T03:36:07.769 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.769 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.769 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.769 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415659+0000", 2023-10-20T03:36:07.769 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6779000000000001e-05 2023-10-20T03:36:07.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.770 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.770 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.770 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415678+0000", 2023-10-20T03:36:07.770 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8635999999999999e-05 2023-10-20T03:36:07.771 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.771 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.771 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.771 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415832+0000", 2023-10-20T03:36:07.771 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015418300000000001 2023-10-20T03:36:07.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.772 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.772 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.772 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.415848+0000", 2023-10-20T03:36:07.772 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5506e-05 2023-10-20T03:36:07.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.773 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.773 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.773 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.435162+0000", 2023-10-20T03:36:07.773 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019313974000000001 2023-10-20T03:36:07.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.774 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.774 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.774 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.435255+0000", 2023-10-20T03:36:07.774 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3350999999999994e-05 2023-10-20T03:36:07.774 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.775 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.775 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.775 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.775 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.775 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 53/47 ECSubWrite(tid=56, reqid=osd.3.0:55, at_version=0'0, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:07.776 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.448564+0000", 2023-10-20T03:36:07.776 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 32.279815714999998, 2023-10-20T03:36:07.776 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014906802, 2023-10-20T03:36:07.776 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.776 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.776 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.777 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.777 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448564+0000", 2023-10-20T03:36:07.777 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.778 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.778 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448564+0000", 2023-10-20T03:36:07.778 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.778 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.779 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.779 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448566+0000", 2023-10-20T03:36:07.779 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8139999999999999e-06 2023-10-20T03:36:07.779 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.779 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.779 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.780 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448593+0000", 2023-10-20T03:36:07.780 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6829e-05 2023-10-20T03:36:07.780 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.780 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.780 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.780 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448629+0000", 2023-10-20T03:36:07.781 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6449000000000003e-05 2023-10-20T03:36:07.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.781 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.781 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.781 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448647+0000", 2023-10-20T03:36:07.781 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7895999999999999e-05 2023-10-20T03:36:07.782 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.782 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.782 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.782 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448792+0000", 2023-10-20T03:36:07.782 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014443400000000001 2023-10-20T03:36:07.783 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.783 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.783 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.783 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.448808+0000", 2023-10-20T03:36:07.783 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6602000000000001e-05 2023-10-20T03:36:07.783 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.784 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.784 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.784 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.463361+0000", 2023-10-20T03:36:07.784 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0145524 2023-10-20T03:36:07.784 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.784 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.785 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.785 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.463471+0000", 2023-10-20T03:36:07.785 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011037800000000001 2023-10-20T03:36:07.785 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.785 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.785 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.786 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.786 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 53/47 ECSubWrite(tid=57, reqid=client.4570.0:408, at_version=53'31, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:07.786 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.449547+0000", 2023-10-20T03:36:07.786 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 32.278833308999999, 2023-10-20T03:36:07.787 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015260123, 2023-10-20T03:36:07.787 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.787 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.787 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.787 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.788 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.788 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.449547+0000", 2023-10-20T03:36:07.788 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.788 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.788 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.788 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.789 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.449547+0000", 2023-10-20T03:36:07.789 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.789 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.789 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.789 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.789 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.449548+0000", 2023-10-20T03:36:07.790 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.767e-06 2023-10-20T03:36:07.790 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.790 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.790 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.790 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.449997+0000", 2023-10-20T03:36:07.791 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00044844900000000002 2023-10-20T03:36:07.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.791 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.791 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.791 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.450025+0000", 2023-10-20T03:36:07.791 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7826000000000001e-05 2023-10-20T03:36:07.792 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.792 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.792 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.792 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.450038+0000", 2023-10-20T03:36:07.792 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3383e-05 2023-10-20T03:36:07.793 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.793 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.793 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.793 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.450200+0000", 2023-10-20T03:36:07.793 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000162349 2023-10-20T03:36:07.793 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.794 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.794 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.794 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.450221+0000", 2023-10-20T03:36:07.794 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1078999999999999e-05 2023-10-20T03:36:07.794 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.795 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.795 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.795 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.464734+0000", 2023-10-20T03:36:07.795 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014512771000000001 2023-10-20T03:36:07.795 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.795 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.796 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.796 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.464807+0000", 2023-10-20T03:36:07.796 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2498999999999998e-05 2023-10-20T03:36:07.796 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.796 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.796 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.797 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.797 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.797 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 53/47 ECSubWrite(tid=58, reqid=client.4570.0:409, at_version=53'32, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:07.797 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.473255+0000", 2023-10-20T03:36:07.797 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 32.255124358000003, 2023-10-20T03:36:07.797 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011057342, 2023-10-20T03:36:07.798 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.798 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.798 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.798 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.798 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.799 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473255+0000", 2023-10-20T03:36:07.799 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.799 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.799 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.799 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.799 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473255+0000", 2023-10-20T03:36:07.800 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.800 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.800 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.800 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.800 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473257+0000", 2023-10-20T03:36:07.801 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9069999999999999e-06 2023-10-20T03:36:07.801 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.801 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.801 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.801 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473620+0000", 2023-10-20T03:36:07.801 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000362667 2023-10-20T03:36:07.802 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.802 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.802 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.802 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473671+0000", 2023-10-20T03:36:07.802 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0828999999999997e-05 2023-10-20T03:36:07.803 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.803 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.803 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.803 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473698+0000", 2023-10-20T03:36:07.803 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6783000000000001e-05 2023-10-20T03:36:07.804 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.804 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.804 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.804 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473830+0000", 2023-10-20T03:36:07.804 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013184000000000001 2023-10-20T03:36:07.804 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.805 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.805 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.805 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.473851+0000", 2023-10-20T03:36:07.805 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1716999999999999e-05 2023-10-20T03:36:07.805 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.806 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.806 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.806 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.484230+0000", 2023-10-20T03:36:07.806 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010379256 2023-10-20T03:36:07.806 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.806 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.807 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.807 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.484313+0000", 2023-10-20T03:36:07.807 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2342999999999994e-05 2023-10-20T03:36:07.807 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.807 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.807 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.808 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.808 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.808 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 54/31 ECSubWrite(tid=65, reqid=client.4570.0:418, at_version=54'32, trim_to=0'0, roll_forward_to=51'31))", 2023-10-20T03:36:07.808 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.459426+0000", 2023-10-20T03:36:07.808 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.268953444000001, 2023-10-20T03:36:07.808 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015193442999999999, 2023-10-20T03:36:07.809 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.809 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.809 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.809 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.809 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.810 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459426+0000", 2023-10-20T03:36:07.810 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.810 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.810 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.810 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.810 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459426+0000", 2023-10-20T03:36:07.811 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.811 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.811 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.811 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.811 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459429+0000", 2023-10-20T03:36:07.811 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2900000000000001e-06 2023-10-20T03:36:07.812 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.812 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.812 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.812 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459850+0000", 2023-10-20T03:36:07.812 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000421266 2023-10-20T03:36:07.813 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.813 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.813 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.813 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459900+0000", 2023-10-20T03:36:07.813 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9914999999999999e-05 2023-10-20T03:36:07.814 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.814 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.814 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.814 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459925+0000", 2023-10-20T03:36:07.814 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5596e-05 2023-10-20T03:36:07.814 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.815 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.815 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.815 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.460072+0000", 2023-10-20T03:36:07.815 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014694400000000001 2023-10-20T03:36:07.815 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.816 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.816 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.816 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.460100+0000", 2023-10-20T03:36:07.816 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7087999999999999e-05 2023-10-20T03:36:07.816 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.816 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.817 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.817 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.474528+0000", 2023-10-20T03:36:07.817 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014428806000000001 2023-10-20T03:36:07.817 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.817 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.817 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.818 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.474620+0000", 2023-10-20T03:36:07.818 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1538000000000003e-05 2023-10-20T03:36:07.818 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.818 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.818 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.819 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.819 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.819 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 54/31 ECSubWrite(tid=66, reqid=client.4570.0:419, at_version=54'33, trim_to=0'0, roll_forward_to=51'31))", 2023-10-20T03:36:07.819 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.476272+0000", 2023-10-20T03:36:07.819 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.252107417000001, 2023-10-20T03:36:07.820 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010573028, 2023-10-20T03:36:07.820 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.820 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.820 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.820 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.820 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.821 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476272+0000", 2023-10-20T03:36:07.821 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.821 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.821 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.821 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.821 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476272+0000", 2023-10-20T03:36:07.822 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.822 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.822 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.822 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.822 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476274+0000", 2023-10-20T03:36:07.823 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9130000000000001e-06 2023-10-20T03:36:07.823 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.823 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.823 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.823 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476566+0000", 2023-10-20T03:36:07.823 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00029126899999999998 2023-10-20T03:36:07.824 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.824 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.824 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.824 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476615+0000", 2023-10-20T03:36:07.824 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9188999999999999e-05 2023-10-20T03:36:07.825 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.825 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.825 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.825 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476640+0000", 2023-10-20T03:36:07.825 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5128999999999999e-05 2023-10-20T03:36:07.825 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.826 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.826 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.826 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476792+0000", 2023-10-20T03:36:07.826 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015181 2023-10-20T03:36:07.826 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.826 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.827 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.827 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.476808+0000", 2023-10-20T03:36:07.827 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5882000000000001e-05 2023-10-20T03:36:07.827 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.827 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.828 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.828 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.486770+0000", 2023-10-20T03:36:07.828 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0099620839999999995 2023-10-20T03:36:07.828 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.828 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.828 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.829 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.486845+0000", 2023-10-20T03:36:07.829 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5752000000000002e-05 2023-10-20T03:36:07.829 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.829 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.829 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.830 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.830 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.830 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 54/31 ECSubWrite(tid=67, reqid=client.4570.0:420, at_version=54'34, trim_to=0'0, roll_forward_to=54'32))", 2023-10-20T03:36:07.830 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.488419+0000", 2023-10-20T03:36:07.830 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.239960833000001, 2023-10-20T03:36:07.830 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0089196740000000007, 2023-10-20T03:36:07.831 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.831 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.831 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.831 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.831 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.832 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488419+0000", 2023-10-20T03:36:07.832 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.832 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.832 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.832 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.832 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488419+0000", 2023-10-20T03:36:07.833 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.833 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.833 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.833 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.833 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488421+0000", 2023-10-20T03:36:07.834 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8899999999999999e-06 2023-10-20T03:36:07.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.834 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.834 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.834 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488517+0000", 2023-10-20T03:36:07.834 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6211000000000004e-05 2023-10-20T03:36:07.835 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.835 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.835 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.835 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488591+0000", 2023-10-20T03:36:07.835 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3850999999999995e-05 2023-10-20T03:36:07.836 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.836 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.836 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.836 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488613+0000", 2023-10-20T03:36:07.836 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1985999999999999e-05 2023-10-20T03:36:07.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.837 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.837 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.837 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488771+0000", 2023-10-20T03:36:07.837 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000158386 2023-10-20T03:36:07.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.838 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.838 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.838 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.488799+0000", 2023-10-20T03:36:07.838 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7503000000000001e-05 2023-10-20T03:36:07.838 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.839 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.839 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.839 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.497130+0000", 2023-10-20T03:36:07.839 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0083314960000000007 2023-10-20T03:36:07.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.839 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.840 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.840 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.497339+0000", 2023-10-20T03:36:07.840 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020835100000000001 2023-10-20T03:36:07.840 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.840 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.840 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.841 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.841 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.841 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 55/47 ECSubWrite(tid=71, reqid=client.4570.0:406, at_version=55'38, trim_to=0'0, roll_forward_to=54'36))", 2023-10-20T03:36:07.841 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:37.473148+0000", 2023-10-20T03:36:07.841 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 30.255232028999998, 2023-10-20T03:36:07.842 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013993848, 2023-10-20T03:36:07.842 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.842 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.842 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.842 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.843 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.843 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473148+0000", 2023-10-20T03:36:07.843 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.843 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.843 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.843 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.844 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473148+0000", 2023-10-20T03:36:07.844 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.844 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.844 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.844 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.845 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473148+0000", 2023-10-20T03:36:07.845 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6600000000000002e-07 2023-10-20T03:36:07.845 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.845 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.845 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.845 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473628+0000", 2023-10-20T03:36:07.846 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00047980800000000002 2023-10-20T03:36:07.846 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.846 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.846 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.846 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473658+0000", 2023-10-20T03:36:07.847 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0159999999999999e-05 2023-10-20T03:36:07.847 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.847 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.847 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.847 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473681+0000", 2023-10-20T03:36:07.847 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2433e-05 2023-10-20T03:36:07.848 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.848 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.848 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.848 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473843+0000", 2023-10-20T03:36:07.848 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016257500000000001 2023-10-20T03:36:07.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.849 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.849 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.849 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.473860+0000", 2023-10-20T03:36:07.849 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6543999999999999e-05 2023-10-20T03:36:07.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.850 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.850 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.850 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.487051+0000", 2023-10-20T03:36:07.850 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013190947 2023-10-20T03:36:07.850 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.851 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.851 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.851 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:37.487142+0000", 2023-10-20T03:36:07.851 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1014999999999995e-05 2023-10-20T03:36:07.851 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.851 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.852 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.852 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.852 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.852 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/47 ECSubWrite(tid=83, reqid=osd.3.0:83, at_version=60'48, trim_to=0'0, roll_forward_to=56'46))", 2023-10-20T03:36:07.852 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.854857+0000", 2023-10-20T03:36:07.853 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 25.873522980000001, 2023-10-20T03:36:07.853 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012200993, 2023-10-20T03:36:07.853 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.853 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.853 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.854 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.854 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.854 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.854857+0000", 2023-10-20T03:36:07.854 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.854 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.854 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.855 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.855 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.854857+0000", 2023-10-20T03:36:07.855 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.855 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.855 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.856 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.856 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.854858+0000", 2023-10-20T03:36:07.856 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9299999999999997e-07 2023-10-20T03:36:07.856 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.856 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.856 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.857 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.854919+0000", 2023-10-20T03:36:07.857 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1148999999999993e-05 2023-10-20T03:36:07.857 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.857 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.857 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.857 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.855027+0000", 2023-10-20T03:36:07.858 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010831399999999999 2023-10-20T03:36:07.858 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.858 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.858 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.858 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.855057+0000", 2023-10-20T03:36:07.859 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9396000000000001e-05 2023-10-20T03:36:07.859 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.859 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.859 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.859 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.855236+0000", 2023-10-20T03:36:07.859 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000179366 2023-10-20T03:36:07.860 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.860 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.860 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.860 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.855267+0000", 2023-10-20T03:36:07.860 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.1133000000000002e-05 2023-10-20T03:36:07.861 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.861 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.861 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.861 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.866928+0000", 2023-10-20T03:36:07.861 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011660841 2023-10-20T03:36:07.861 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.862 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.862 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.862 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.867058+0000", 2023-10-20T03:36:07.862 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000130001 2023-10-20T03:36:07.862 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.863 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.863 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.863 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.863 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.863 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/47 ECSubWrite(tid=84, reqid=osd.3.0:84, at_version=60'50, trim_to=0'0, roll_forward_to=56'46))", 2023-10-20T03:36:07.863 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.857484+0000", 2023-10-20T03:36:07.864 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 25.870895854, 2023-10-20T03:36:07.864 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012226756, 2023-10-20T03:36:07.864 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.864 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.864 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.865 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.865 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.865 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857484+0000", 2023-10-20T03:36:07.865 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.865 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.865 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.866 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.866 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857484+0000", 2023-10-20T03:36:07.866 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.866 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.866 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.867 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.867 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857486+0000", 2023-10-20T03:36:07.867 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.04e-06 2023-10-20T03:36:07.867 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.867 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.868 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.868 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857540+0000", 2023-10-20T03:36:07.868 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4339000000000003e-05 2023-10-20T03:36:07.868 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.868 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.868 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.869 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857575+0000", 2023-10-20T03:36:07.869 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4555999999999999e-05 2023-10-20T03:36:07.869 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.869 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.869 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.870 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857598+0000", 2023-10-20T03:36:07.870 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3193e-05 2023-10-20T03:36:07.870 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.870 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.870 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.870 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.864569+0000", 2023-10-20T03:36:07.871 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0069706580000000002 2023-10-20T03:36:07.871 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.871 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.871 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.871 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.864611+0000", 2023-10-20T03:36:07.872 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1909000000000001e-05 2023-10-20T03:36:07.872 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.872 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.872 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.872 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.869616+0000", 2023-10-20T03:36:07.872 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00500557 2023-10-20T03:36:07.873 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.873 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.873 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.873 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.869711+0000", 2023-10-20T03:36:07.873 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4491e-05 2023-10-20T03:36:07.874 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.874 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.874 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.874 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.874 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.874 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 60/47 ECSubWrite(tid=87, reqid=osd.3.0:87, at_version=60'54, trim_to=0'0, roll_forward_to=60'50))", 2023-10-20T03:36:07.875 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.888724+0000", 2023-10-20T03:36:07.875 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 25.839655459999999, 2023-10-20T03:36:07.875 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010164182000000001, 2023-10-20T03:36:07.875 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.875 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.876 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.876 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.876 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.876 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888724+0000", 2023-10-20T03:36:07.876 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.876 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.877 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.877 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.877 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888724+0000", 2023-10-20T03:36:07.877 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.877 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.878 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.878 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.878 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888725+0000", 2023-10-20T03:36:07.878 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0240000000000001e-06 2023-10-20T03:36:07.878 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.879 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.879 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.879 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888808+0000", 2023-10-20T03:36:07.879 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2757999999999996e-05 2023-10-20T03:36:07.879 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.880 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.880 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.880 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888836+0000", 2023-10-20T03:36:07.880 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7572e-05 2023-10-20T03:36:07.880 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.880 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.881 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.881 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888855+0000", 2023-10-20T03:36:07.881 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.961e-05 2023-10-20T03:36:07.881 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.881 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.882 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.882 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889902+0000", 2023-10-20T03:36:07.882 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0010469730000000001 2023-10-20T03:36:07.882 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.882 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.883 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.883 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889928+0000", 2023-10-20T03:36:07.883 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5916000000000001e-05 2023-10-20T03:36:07.883 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.883 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.883 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.884 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.898832+0000", 2023-10-20T03:36:07.884 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0089040820000000007 2023-10-20T03:36:07.884 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.884 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.884 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.885 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.898889+0000", 2023-10-20T03:36:07.885 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6246999999999999e-05 2023-10-20T03:36:07.885 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.885 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.885 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.885 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.886 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.886 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs2 65/47 ECSubWrite(tid=93, reqid=osd.3.0:93, at_version=65'60, trim_to=0'0, roll_forward_to=62'58))", 2023-10-20T03:36:07.886 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:46.896396+0000", 2023-10-20T03:36:07.886 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 20.831984288000001, 2023-10-20T03:36:07.886 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.008527817, 2023-10-20T03:36:07.887 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.887 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.887 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.887 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.887 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.887 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:46.896396+0000", 2023-10-20T03:36:07.888 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.888 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.888 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.888 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.888 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:46.896396+0000", 2023-10-20T03:36:07.888 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.889 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.889 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.889 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.889 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:46.896396+0000", 2023-10-20T03:36:07.889 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1999999999999998e-07 2023-10-20T03:36:07.890 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.890 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.890 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.890 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:46.896453+0000", 2023-10-20T03:36:07.890 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6184999999999998e-05 2023-10-20T03:36:07.890 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.891 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.891 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.891 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:46.896488+0000", 2023-10-20T03:36:07.891 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5876e-05 2023-10-20T03:36:07.891 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.892 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.892 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.892 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:46.896514+0000", 2023-10-20T03:36:07.892 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5666000000000001e-05 2023-10-20T03:36:07.892 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.892 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.893 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.893 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:46.896704+0000", 2023-10-20T03:36:07.893 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018981 2023-10-20T03:36:07.893 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.893 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.894 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.894 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:46.896735+0000", 2023-10-20T03:36:07.894 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0621999999999999e-05 2023-10-20T03:36:07.894 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.894 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.894 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.895 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:46.904804+0000", 2023-10-20T03:36:07.895 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0080694860000000007 2023-10-20T03:36:07.895 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.895 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.895 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.896 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:46.904923+0000", 2023-10-20T03:36:07.896 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011935200000000001 2023-10-20T03:36:07.896 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.896 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.896 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.896 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.897 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.897 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 69/31 ECSubWrite(tid=95, reqid=client.4570.0:552, at_version=69'41, trim_to=0'0, roll_forward_to=56'39))", 2023-10-20T03:36:07.897 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:50.954533+0000", 2023-10-20T03:36:07.897 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 16.773846410000001, 2023-10-20T03:36:07.897 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.031030447999999999, 2023-10-20T03:36:07.898 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.898 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.898 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.898 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.898 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.899 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.954533+0000", 2023-10-20T03:36:07.899 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.899 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.899 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.899 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.899 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.954533+0000", 2023-10-20T03:36:07.900 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.900 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.900 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.900 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.900 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.954534+0000", 2023-10-20T03:36:07.900 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0299999999999999e-07 2023-10-20T03:36:07.901 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.901 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.901 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.901 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.955698+0000", 2023-10-20T03:36:07.901 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0011639249999999999 2023-10-20T03:36:07.902 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.902 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.902 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.902 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.955744+0000", 2023-10-20T03:36:07.902 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6285000000000002e-05 2023-10-20T03:36:07.903 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.903 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.903 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.903 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.955773+0000", 2023-10-20T03:36:07.903 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8459999999999999e-05 2023-10-20T03:36:07.904 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.904 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.904 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.904 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.955951+0000", 2023-10-20T03:36:07.904 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017856999999999999 2023-10-20T03:36:07.904 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.905 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.905 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.905 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.955978+0000", 2023-10-20T03:36:07.905 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6588999999999999e-05 2023-10-20T03:36:07.905 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.906 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.906 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.906 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.985526+0000", 2023-10-20T03:36:07.906 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029548013000000001 2023-10-20T03:36:07.906 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.907 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.907 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.907 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.985564+0000", 2023-10-20T03:36:07.907 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8003000000000002e-05 2023-10-20T03:36:07.907 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.908 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.908 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.908 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.908 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.908 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.as5 71/56 ECSubWrite(tid=21, reqid=client.4570.0:598, at_version=71'27, trim_to=0'0, roll_forward_to=71'26))", 2023-10-20T03:36:07.908 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.214007+0000", 2023-10-20T03:36:07.909 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.514373213000001, 2023-10-20T03:36:07.909 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014267537, 2023-10-20T03:36:07.909 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.909 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.909 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.910 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.910 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.910 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.214007+0000", 2023-10-20T03:36:07.910 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.910 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.911 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.911 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.911 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.214007+0000", 2023-10-20T03:36:07.911 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.911 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.911 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.912 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.912 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.214010+0000", 2023-10-20T03:36:07.912 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5329999999999999e-06 2023-10-20T03:36:07.912 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.912 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.913 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.913 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.214378+0000", 2023-10-20T03:36:07.913 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00036825399999999999 2023-10-20T03:36:07.913 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.913 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.913 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.914 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.214412+0000", 2023-10-20T03:36:07.914 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3281999999999997e-05 2023-10-20T03:36:07.914 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.914 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.914 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.915 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.214436+0000", 2023-10-20T03:36:07.915 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3827000000000001e-05 2023-10-20T03:36:07.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.915 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.915 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.916 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.214568+0000", 2023-10-20T03:36:07.916 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000132681 2023-10-20T03:36:07.916 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.916 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.916 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.917 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.214586+0000", 2023-10-20T03:36:07.917 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8139e-05 2023-10-20T03:36:07.917 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.917 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.917 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.917 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.228164+0000", 2023-10-20T03:36:07.918 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013577933 2023-10-20T03:36:07.918 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.918 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.918 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.918 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.228274+0000", 2023-10-20T03:36:07.919 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000109888 2023-10-20T03:36:07.919 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.919 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.919 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.919 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.920 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.920 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.as5 71/56 ECSubWrite(tid=22, reqid=client.4570.0:599, at_version=71'28, trim_to=0'0, roll_forward_to=71'26))", 2023-10-20T03:36:07.920 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.231929+0000", 2023-10-20T03:36:07.920 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 14.496451220999999, 2023-10-20T03:36:07.920 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011102707, 2023-10-20T03:36:07.920 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.921 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.921 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.921 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.921 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.921 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.231929+0000", 2023-10-20T03:36:07.922 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.922 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.922 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.922 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.922 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.231929+0000", 2023-10-20T03:36:07.922 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.923 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.923 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.923 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.923 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.231930+0000", 2023-10-20T03:36:07.923 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5200000000000001e-06 2023-10-20T03:36:07.924 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.924 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.924 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.924 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.232101+0000", 2023-10-20T03:36:07.924 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000170624 2023-10-20T03:36:07.925 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.925 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.925 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.925 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.232130+0000", 2023-10-20T03:36:07.925 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8937000000000001e-05 2023-10-20T03:36:07.925 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.926 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.926 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.926 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.232147+0000", 2023-10-20T03:36:07.926 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7489000000000001e-05 2023-10-20T03:36:07.926 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.927 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.927 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.927 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.232300+0000", 2023-10-20T03:36:07.927 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015243400000000001 2023-10-20T03:36:07.927 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.928 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.928 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.928 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.232318+0000", 2023-10-20T03:36:07.928 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8539999999999999e-05 2023-10-20T03:36:07.928 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.928 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.929 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.929 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.242974+0000", 2023-10-20T03:36:07.929 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010655394 2023-10-20T03:36:07.929 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.929 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.929 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.930 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.243031+0000", 2023-10-20T03:36:07.930 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7769000000000002e-05 2023-10-20T03:36:07.930 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.930 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.930 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.930 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.931 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.931 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s5 75/31 ECSubWrite(tid=97, reqid=client.4570.0:628, at_version=75'43, trim_to=0'0, roll_forward_to=69'41))", 2023-10-20T03:36:07.931 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:57.036941+0000", 2023-10-20T03:36:07.931 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 10.691438985, 2023-10-20T03:36:07.931 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.042387374999999998, 2023-10-20T03:36:07.932 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.932 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.932 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.932 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.932 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.932 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.036941+0000", 2023-10-20T03:36:07.933 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.933 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.933 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.933 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.933 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.036941+0000", 2023-10-20T03:36:07.933 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.934 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.934 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.934 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.934 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.036941+0000", 2023-10-20T03:36:07.934 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5999999999999999e-07 2023-10-20T03:36:07.935 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.935 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.935 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.935 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.037746+0000", 2023-10-20T03:36:07.935 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00080458500000000002 2023-10-20T03:36:07.935 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.936 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.936 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.936 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.037777+0000", 2023-10-20T03:36:07.936 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0775000000000003e-05 2023-10-20T03:36:07.936 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.936 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.937 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.937 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.037798+0000", 2023-10-20T03:36:07.937 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1296000000000001e-05 2023-10-20T03:36:07.937 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.937 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.937 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.938 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.037939+0000", 2023-10-20T03:36:07.938 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014116199999999999 2023-10-20T03:36:07.938 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.938 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.938 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.939 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.037964+0000", 2023-10-20T03:36:07.939 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4685000000000002e-05 2023-10-20T03:36:07.939 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.939 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.939 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.940 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.079253+0000", 2023-10-20T03:36:07.940 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.041289160999999998 2023-10-20T03:36:07.940 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.940 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.940 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.940 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.079328+0000", 2023-10-20T03:36:07.941 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5251000000000002e-05 2023-10-20T03:36:07.941 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.941 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.941 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.941 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.942 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.942 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.as5 82/56 ECSubWrite(tid=36, reqid=client.4570.0:664, at_version=82'39, trim_to=0'0, roll_forward_to=81'35))", 2023-10-20T03:36:07.942 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:36:03.785863+0000", 2023-10-20T03:36:07.942 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 3.9425170629999999, 2023-10-20T03:36:07.942 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0090346509999999994, 2023-10-20T03:36:07.942 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:07.943 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:07.943 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:07.943 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.943 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:07.943 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:03.785863+0000", 2023-10-20T03:36:07.944 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.944 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.944 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.944 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:07.944 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:03.785863+0000", 2023-10-20T03:36:07.945 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:07.945 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.945 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.945 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:07.945 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:03.785865+0000", 2023-10-20T03:36:07.946 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8929999999999999e-06 2023-10-20T03:36:07.946 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.946 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.946 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:07.946 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:03.785922+0000", 2023-10-20T03:36:07.947 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7383999999999998e-05 2023-10-20T03:36:07.947 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.947 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.947 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:07.947 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:03.785957+0000", 2023-10-20T03:36:07.947 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5045000000000001e-05 2023-10-20T03:36:07.948 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.948 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.948 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:07.948 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:03.785981+0000", 2023-10-20T03:36:07.948 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4256e-05 2023-10-20T03:36:07.949 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.949 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.949 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:07.949 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:03.789803+0000", 2023-10-20T03:36:07.949 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0038217250000000002 2023-10-20T03:36:07.949 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.950 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.950 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:07.950 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:03.789835+0000", 2023-10-20T03:36:07.950 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.1908e-05 2023-10-20T03:36:07.950 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.951 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.951 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:07.951 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:03.794814+0000", 2023-10-20T03:36:07.951 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0049788569999999997 2023-10-20T03:36:07.951 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:07.951 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:07.952 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:07.952 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:03.794897+0000", 2023-10-20T03:36:07.952 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3583000000000003e-05 2023-10-20T03:36:07.952 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.952 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.953 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.953 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:07.953 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:07.953 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:07.954 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-10-20T03:36:07.955 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:07.949+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:07.956 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-10-20T03:36:08.056 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:36:08.052+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:36:08.065 INFO:tasks.rados.rados.0.smithi170.stdout:294: done (1 left) 2023-10-20T03:36:08.065 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 40 current snap is 26 2023-10-20T03:36:08.065 INFO:tasks.rados.rados.0.smithi170.stdout:295: seq_num 127 ranges {2555904=589824,3145728=753664,3899392=475136,4374528=327680} 2023-10-20T03:36:08.086 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:08.086 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:36:08.087 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:36:08.087 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:08.105 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-10-20T03:36:08.117 INFO:tasks.rados.rados.0.smithi170.stdout:295: writing smithi17039300-40 from 2555904 to 3145728 tid 1 2023-10-20T03:36:08.129 INFO:tasks.rados.rados.0.smithi170.stdout:295: writing smithi17039300-40 from 3145728 to 3899392 tid 2 2023-10-20T03:36:08.135 INFO:tasks.rados.rados.0.smithi170.stdout:295: writing smithi17039300-40 from 3899392 to 4374528 tid 3 2023-10-20T03:36:08.138 INFO:tasks.rados.rados.0.smithi170.stdout:295: writing smithi17039300-40 from 4374528 to 4702208 tid 4 2023-10-20T03:36:08.138 INFO:tasks.rados.rados.0.smithi170.stdout:296: snap_remove snap 25 2023-10-20T03:36:08.156 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:08.152+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:08.226 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:08.227 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:36:08.227 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:36:08.227 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:36:08.227 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:08.240 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-10-20T03:36:08.258 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:08.256+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:08.359 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:08.356+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:08.401 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:08.402 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:36:08.402 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:36:08.402 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:36:08.402 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.403 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s5 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:36:08.403 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.356693+0000", 2023-10-20T03:36:08.403 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 43.041626092000001, 2023-10-20T03:36:08.403 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010267518, 2023-10-20T03:36:08.403 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.404 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.404 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.404 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.404 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.404 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356693+0000", 2023-10-20T03:36:08.405 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.405 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.405 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.405 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.405 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356693+0000", 2023-10-20T03:36:08.406 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.406 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.406 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.406 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356694+0000", 2023-10-20T03:36:08.407 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5000000000000002e-07 2023-10-20T03:36:08.407 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.407 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.407 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.407 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357174+0000", 2023-10-20T03:36:08.407 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047994599999999998 2023-10-20T03:36:08.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.408 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.408 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.408 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357194+0000", 2023-10-20T03:36:08.408 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.978e-05 2023-10-20T03:36:08.409 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.409 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.409 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.409 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357209+0000", 2023-10-20T03:36:08.409 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5506e-05 2023-10-20T03:36:08.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.410 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.410 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.410 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357336+0000", 2023-10-20T03:36:08.410 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012640000000000001 2023-10-20T03:36:08.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.411 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.411 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.411 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357353+0000", 2023-10-20T03:36:08.411 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6867e-05 2023-10-20T03:36:08.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.412 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.412 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.412 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.366907+0000", 2023-10-20T03:36:08.412 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.009554412 2023-10-20T03:36:08.412 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.413 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.413 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.413 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.366961+0000", 2023-10-20T03:36:08.413 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3956999999999999e-05 2023-10-20T03:36:08.413 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.414 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.414 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.414 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.414 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.414 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:331 3.cs0 3:3382ea17:::smithi17039300-35:head [rollback a] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:08.415 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.371212+0000", 2023-10-20T03:36:08.415 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 42.027107706000002, 2023-10-20T03:36:08.415 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010542586, 2023-10-20T03:36:08.415 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.415 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:08.416 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:08.416 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:08.416 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:08.416 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 331 2023-10-20T03:36:08.416 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.416 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.417 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.417 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.417 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371212+0000", 2023-10-20T03:36:08.417 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.417 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.418 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.418 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.418 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371212+0000", 2023-10-20T03:36:08.418 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.418 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.419 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.419 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.419 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371213+0000", 2023-10-20T03:36:08.419 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1999999999999998e-07 2023-10-20T03:36:08.419 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.420 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.420 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.420 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371219+0000", 2023-10-20T03:36:08.420 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2600000000000002e-06 2023-10-20T03:36:08.420 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.421 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.421 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.421 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371233+0000", 2023-10-20T03:36:08.421 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3816e-05 2023-10-20T03:36:08.421 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.422 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.422 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.422 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371241+0000", 2023-10-20T03:36:08.422 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5569999999999996e-06 2023-10-20T03:36:08.422 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.422 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.423 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.423 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.372719+0000", 2023-10-20T03:36:08.423 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014780539999999999 2023-10-20T03:36:08.423 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.423 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.424 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:08.424 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.372777+0000", 2023-10-20T03:36:08.424 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7485999999999998e-05 2023-10-20T03:36:08.424 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.424 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.425 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.425 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.373138+0000", 2023-10-20T03:36:08.425 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000361679 2023-10-20T03:36:08.425 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.425 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.425 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.426 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.376244+0000", 2023-10-20T03:36:08.426 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003105652 2023-10-20T03:36:08.426 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.426 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.426 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:08.427 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381631+0000", 2023-10-20T03:36:08.427 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053871179999999998 2023-10-20T03:36:08.427 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.427 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.427 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.428 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381754+0000", 2023-10-20T03:36:08.428 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000123144 2023-10-20T03:36:08.428 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.428 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.428 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.429 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.429 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.429 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:332 3.cs0 3:3382ea17:::smithi17039300-35:head [append 0~0] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:08.429 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.371334+0000", 2023-10-20T03:36:08.430 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 42.026984988999999, 2023-10-20T03:36:08.430 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010711231, 2023-10-20T03:36:08.430 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.430 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:08.430 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:08.431 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:08.431 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:08.431 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 332 2023-10-20T03:36:08.431 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.431 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.431 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.432 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.432 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371334+0000", 2023-10-20T03:36:08.432 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.432 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.432 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.433 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.433 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371334+0000", 2023-10-20T03:36:08.433 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.433 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.433 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.433 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.434 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371335+0000", 2023-10-20T03:36:08.434 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.47e-07 2023-10-20T03:36:08.434 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.434 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.434 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.435 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371340+0000", 2023-10-20T03:36:08.435 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5800000000000002e-06 2023-10-20T03:36:08.435 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.435 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.435 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.435 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371366+0000", 2023-10-20T03:36:08.436 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5639e-05 2023-10-20T03:36:08.436 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.436 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.436 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.436 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371379+0000", 2023-10-20T03:36:08.437 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.346e-05 2023-10-20T03:36:08.437 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.437 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.437 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.437 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.373793+0000", 2023-10-20T03:36:08.438 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024143049999999998 2023-10-20T03:36:08.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.438 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.438 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:08.438 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.373855+0000", 2023-10-20T03:36:08.438 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1277000000000003e-05 2023-10-20T03:36:08.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.439 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.439 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.439 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.374199+0000", 2023-10-20T03:36:08.439 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00034445500000000002 2023-10-20T03:36:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.376260+0000", 2023-10-20T03:36:08.440 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020611050000000001 2023-10-20T03:36:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381945+0000", 2023-10-20T03:36:08.441 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056847690000000001 2023-10-20T03:36:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.382046+0000", 2023-10-20T03:36:08.442 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000100794 2023-10-20T03:36:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.443 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.444 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:08.444 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.387979+0000", 2023-10-20T03:36:08.444 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 42.010340749000001, 2023-10-20T03:36:08.444 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018548599999999998, 2023-10-20T03:36:08.444 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.445 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387979+0000", 2023-10-20T03:36:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.446 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387979+0000", 2023-10-20T03:36:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.447 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387979+0000", 2023-10-20T03:36:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6000000000000003e-07 2023-10-20T03:36:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.448 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388153+0000", 2023-10-20T03:36:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017364600000000001 2023-10-20T03:36:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388173+0000", 2023-10-20T03:36:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9961000000000001e-05 2023-10-20T03:36:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.450 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388186+0000", 2023-10-20T03:36:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2403e-05 2023-10-20T03:36:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.451 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388293+0000", 2023-10-20T03:36:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010739300000000001 2023-10-20T03:36:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388307+0000", 2023-10-20T03:36:08.452 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3773e-05 2023-10-20T03:36:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406469+0000", 2023-10-20T03:36:08.453 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018162098000000002 2023-10-20T03:36:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406527+0000", 2023-10-20T03:36:08.454 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8566000000000001e-05 2023-10-20T03:36:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.455 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.394884+0000", 2023-10-20T03:36:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 42.003435650999997, 2023-10-20T03:36:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011674960999999999, 2023-10-20T03:36:08.456 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394884+0000", 2023-10-20T03:36:08.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394884+0000", 2023-10-20T03:36:08.458 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394886+0000", 2023-10-20T03:36:08.459 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0269999999999998e-06 2023-10-20T03:36:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395153+0000", 2023-10-20T03:36:08.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026702300000000002 2023-10-20T03:36:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395187+0000", 2023-10-20T03:36:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3738999999999997e-05 2023-10-20T03:36:08.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395211+0000", 2023-10-20T03:36:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4859e-05 2023-10-20T03:36:08.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395301+0000", 2023-10-20T03:36:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9553000000000001e-05 2023-10-20T03:36:08.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395310+0000", 2023-10-20T03:36:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9439999999999994e-06 2023-10-20T03:36:08.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406535+0000", 2023-10-20T03:36:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011225153999999999 2023-10-20T03:36:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.465 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406559+0000", 2023-10-20T03:36:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3662000000000001e-05 2023-10-20T03:36:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.466 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs0 48/47 ECSubWrite(tid=115, reqid=osd.13.0:115, at_version=48'27, trim_to=0'0, roll_forward_to=0'0))", 2023-10-20T03:36:08.467 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.389413+0000", 2023-10-20T03:36:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 38.008906189000001, 2023-10-20T03:36:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0097928949999999994, 2023-10-20T03:36:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.468 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389413+0000", 2023-10-20T03:36:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389413+0000", 2023-10-20T03:36:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.470 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389416+0000", 2023-10-20T03:36:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2170000000000001e-06 2023-10-20T03:36:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389474+0000", 2023-10-20T03:36:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8931999999999998e-05 2023-10-20T03:36:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389502+0000", 2023-10-20T03:36:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7538999999999998e-05 2023-10-20T03:36:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389531+0000", 2023-10-20T03:36:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9172000000000001e-05 2023-10-20T03:36:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389702+0000", 2023-10-20T03:36:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017064000000000001 2023-10-20T03:36:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389767+0000", 2023-10-20T03:36:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5297999999999997e-05 2023-10-20T03:36:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.399101+0000", 2023-10-20T03:36:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093336600000000006 2023-10-20T03:36:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.399206+0000", 2023-10-20T03:36:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000105437 2023-10-20T03:36:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:36:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.392018+0000", 2023-10-20T03:36:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 38.006301710000002, 2023-10-20T03:36:08.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037305552999999998, 2023-10-20T03:36:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392018+0000", 2023-10-20T03:36:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392018+0000", 2023-10-20T03:36:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392020+0000", 2023-10-20T03:36:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2029999999999999e-06 2023-10-20T03:36:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392961+0000", 2023-10-20T03:36:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00094143000000000004 2023-10-20T03:36:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392993+0000", 2023-10-20T03:36:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1795999999999998e-05 2023-10-20T03:36:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.485 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393083+0000", 2023-10-20T03:36:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9584000000000002e-05 2023-10-20T03:36:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393389+0000", 2023-10-20T03:36:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00030593699999999999 2023-10-20T03:36:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.487 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393416+0000", 2023-10-20T03:36:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7639000000000001e-05 2023-10-20T03:36:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429229+0000", 2023-10-20T03:36:08.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.035812384000000003 2023-10-20T03:36:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429323+0000", 2023-10-20T03:36:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4580000000000006e-05 2023-10-20T03:36:08.489 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.490 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.490 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.490 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.fs0 48/47 ECSubWrite(tid=35, reqid=osd.3.0:34, at_version=0'0, trim_to=0'0, roll_forward_to=48'21))", 2023-10-20T03:36:08.491 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.393341+0000", 2023-10-20T03:36:08.491 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 38.004977992999997, 2023-10-20T03:36:08.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092276119999999996, 2023-10-20T03:36:08.491 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.491 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.492 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393341+0000", 2023-10-20T03:36:08.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393341+0000", 2023-10-20T03:36:08.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393343+0000", 2023-10-20T03:36:08.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5740000000000001e-06 2023-10-20T03:36:08.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.495 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.495 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393374+0000", 2023-10-20T03:36:08.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0639000000000003e-05 2023-10-20T03:36:08.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393404+0000", 2023-10-20T03:36:08.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0521999999999997e-05 2023-10-20T03:36:08.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.497 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393425+0000", 2023-10-20T03:36:08.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0993e-05 2023-10-20T03:36:08.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.498 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.498 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393584+0000", 2023-10-20T03:36:08.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015895500000000001 2023-10-20T03:36:08.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393611+0000", 2023-10-20T03:36:08.499 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6361999999999999e-05 2023-10-20T03:36:08.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.402462+0000", 2023-10-20T03:36:08.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0088519600000000007 2023-10-20T03:36:08.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.501 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.501 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.402569+0000", 2023-10-20T03:36:08.501 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000106607 2023-10-20T03:36:08.501 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.502 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.502 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.502 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 49/31 ECSubWrite(tid=63, reqid=osd.12.0:63, at_version=49'27, trim_to=0'0, roll_forward_to=48'23))", 2023-10-20T03:36:08.503 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.407639+0000", 2023-10-20T03:36:08.503 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 36.990680924000003, 2023-10-20T03:36:08.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0082579339999999998, 2023-10-20T03:36:08.503 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.503 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.504 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407639+0000", 2023-10-20T03:36:08.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.505 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.505 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407639+0000", 2023-10-20T03:36:08.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.506 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.506 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.506 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407639+0000", 2023-10-20T03:36:08.506 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4799999999999998e-07 2023-10-20T03:36:08.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.507 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.507 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407669+0000", 2023-10-20T03:36:08.507 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0235000000000001e-05 2023-10-20T03:36:08.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.508 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.508 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.508 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407681+0000", 2023-10-20T03:36:08.508 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2129e-05 2023-10-20T03:36:08.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.509 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.509 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.509 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.407710+0000", 2023-10-20T03:36:08.509 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8816999999999999e-05 2023-10-20T03:36:08.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.510 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.510 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409705+0000", 2023-10-20T03:36:08.510 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019947340000000002 2023-10-20T03:36:08.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.510 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.511 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.511 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.409718+0000", 2023-10-20T03:36:08.511 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2618e-05 2023-10-20T03:36:08.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.511 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.512 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.512 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415856+0000", 2023-10-20T03:36:08.512 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0061379379999999999 2023-10-20T03:36:08.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.512 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.513 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.513 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415896+0000", 2023-10-20T03:36:08.513 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0914999999999997e-05 2023-10-20T03:36:08.513 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.513 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.513 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.514 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.514 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s3 49/31 ECSubWrite(tid=25, reqid=osd.10.0:24, at_version=0'0, trim_to=0'0, roll_forward_to=49'15))", 2023-10-20T03:36:08.514 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.408468+0000", 2023-10-20T03:36:08.514 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 36.989851002999998, 2023-10-20T03:36:08.515 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014029283, 2023-10-20T03:36:08.515 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.515 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.515 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.515 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.516 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.516 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408468+0000", 2023-10-20T03:36:08.516 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.517 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.517 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408468+0000", 2023-10-20T03:36:08.517 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.517 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.518 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.518 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408470+0000", 2023-10-20T03:36:08.518 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1680000000000001e-06 2023-10-20T03:36:08.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.518 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.519 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408486+0000", 2023-10-20T03:36:08.519 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6209999999999999e-05 2023-10-20T03:36:08.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.519 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.519 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.520 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408502+0000", 2023-10-20T03:36:08.520 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6351e-05 2023-10-20T03:36:08.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.520 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.520 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408514+0000", 2023-10-20T03:36:08.521 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1364e-05 2023-10-20T03:36:08.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.521 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.521 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.521 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408748+0000", 2023-10-20T03:36:08.522 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000234088 2023-10-20T03:36:08.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.522 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.522 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.522 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408763+0000", 2023-10-20T03:36:08.523 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5586999999999998e-05 2023-10-20T03:36:08.523 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.523 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.523 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.523 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.422444+0000", 2023-10-20T03:36:08.524 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013680701999999999 2023-10-20T03:36:08.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.524 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.524 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.422498+0000", 2023-10-20T03:36:08.525 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3813000000000001e-05 2023-10-20T03:36:08.525 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.525 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.525 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.526 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.526 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.fs0 53/47 ECSubWrite(tid=57, reqid=client.4570.0:408, at_version=53'31, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:08.526 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:35.448532+0000", 2023-10-20T03:36:08.526 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 32.949787594, 2023-10-20T03:36:08.526 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012712822, 2023-10-20T03:36:08.527 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.527 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.527 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.527 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.527 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.527 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.448532+0000", 2023-10-20T03:36:08.528 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.528 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.528 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.528 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.448532+0000", 2023-10-20T03:36:08.529 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.529 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.529 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.529 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.448532+0000", 2023-10-20T03:36:08.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2799999999999996e-07 2023-10-20T03:36:08.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.530 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.530 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.530 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449112+0000", 2023-10-20T03:36:08.530 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00057979400000000001 2023-10-20T03:36:08.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.531 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.531 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449124+0000", 2023-10-20T03:36:08.531 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1613999999999999e-05 2023-10-20T03:36:08.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.532 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.532 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.532 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449129+0000", 2023-10-20T03:36:08.532 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6049999999999997e-06 2023-10-20T03:36:08.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.533 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.533 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449245+0000", 2023-10-20T03:36:08.533 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011568199999999999 2023-10-20T03:36:08.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.534 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.534 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.534 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449261+0000", 2023-10-20T03:36:08.534 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6376999999999999e-05 2023-10-20T03:36:08.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.535 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.535 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.535 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.461172+0000", 2023-10-20T03:36:08.535 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01191096 2023-10-20T03:36:08.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.536 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.536 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.536 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.461245+0000", 2023-10-20T03:36:08.536 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2261999999999994e-05 2023-10-20T03:36:08.536 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.536 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.537 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.537 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.537 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.fs0 53/47 ECSubWrite(tid=58, reqid=client.4570.0:409, at_version=53'32, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:08.537 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:35.472456+0000", 2023-10-20T03:36:08.538 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 32.925863059000001, 2023-10-20T03:36:08.538 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011134987000000001, 2023-10-20T03:36:08.538 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.538 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.538 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.539 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.539 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472456+0000", 2023-10-20T03:36:08.539 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.539 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.540 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.540 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472456+0000", 2023-10-20T03:36:08.540 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.540 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.541 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472457+0000", 2023-10-20T03:36:08.541 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7999999999999996e-07 2023-10-20T03:36:08.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.541 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.541 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.542 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472728+0000", 2023-10-20T03:36:08.542 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00027096800000000001 2023-10-20T03:36:08.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.542 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.543 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472738+0000", 2023-10-20T03:36:08.543 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0495e-05 2023-10-20T03:36:08.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.543 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.543 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.544 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472744+0000", 2023-10-20T03:36:08.544 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0410000000000003e-06 2023-10-20T03:36:08.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.544 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.544 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472867+0000", 2023-10-20T03:36:08.545 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012291400000000001 2023-10-20T03:36:08.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.545 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.545 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.545 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472883+0000", 2023-10-20T03:36:08.546 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5628e-05 2023-10-20T03:36:08.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.546 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.546 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.546 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.483517+0000", 2023-10-20T03:36:08.547 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010634423 2023-10-20T03:36:08.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.547 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.547 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.547 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.483591+0000", 2023-10-20T03:36:08.548 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4037999999999998e-05 2023-10-20T03:36:08.548 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.548 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.548 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.548 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.549 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 60/56 ECSubWrite(tid=27, reqid=osd.1.0:27, at_version=60'45, trim_to=0'0, roll_forward_to=57'43))", 2023-10-20T03:36:08.549 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.831061+0000", 2023-10-20T03:36:08.549 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.567258957, 2023-10-20T03:36:08.549 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.008194425, 2023-10-20T03:36:08.549 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.550 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.550 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.550 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.550 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.550 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831061+0000", 2023-10-20T03:36:08.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.551 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.551 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.551 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831061+0000", 2023-10-20T03:36:08.551 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.552 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.552 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831061+0000", 2023-10-20T03:36:08.552 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3799999999999998e-07 2023-10-20T03:36:08.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.553 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.553 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.553 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831106+0000", 2023-10-20T03:36:08.553 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.4922000000000001e-05 2023-10-20T03:36:08.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.554 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.554 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.554 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831127+0000", 2023-10-20T03:36:08.554 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1013999999999999e-05 2023-10-20T03:36:08.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.555 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.555 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.555 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831144+0000", 2023-10-20T03:36:08.555 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6935e-05 2023-10-20T03:36:08.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.556 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.556 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.556 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831295+0000", 2023-10-20T03:36:08.556 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015154599999999999 2023-10-20T03:36:08.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.557 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.557 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.831319+0000", 2023-10-20T03:36:08.557 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3764000000000002e-05 2023-10-20T03:36:08.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.558 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.558 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.558 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.839123+0000", 2023-10-20T03:36:08.558 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.007803918 2023-10-20T03:36:08.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.559 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.559 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.559 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.839255+0000", 2023-10-20T03:36:08.559 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013198800000000001 2023-10-20T03:36:08.559 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.560 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.560 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.560 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.560 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.7s6 60/31 ECSubWrite(tid=200, reqid=client.4570.0:459, at_version=60'9, trim_to=0'0, roll_forward_to=38'7))", 2023-10-20T03:36:08.561 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.897194+0000", 2023-10-20T03:36:08.561 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.501125743999999, 2023-10-20T03:36:08.561 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083457080000000003, 2023-10-20T03:36:08.561 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.561 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.561 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.562 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.562 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.562 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897194+0000", 2023-10-20T03:36:08.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.563 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.563 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897194+0000", 2023-10-20T03:36:08.563 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.564 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.564 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897194+0000", 2023-10-20T03:36:08.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4399999999999999e-07 2023-10-20T03:36:08.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.565 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897530+0000", 2023-10-20T03:36:08.565 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00033525200000000002 2023-10-20T03:36:08.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897545+0000", 2023-10-20T03:36:08.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5299999999999999e-05 2023-10-20T03:36:08.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897553+0000", 2023-10-20T03:36:08.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.7940000000000003e-06 2023-10-20T03:36:08.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897629+0000", 2023-10-20T03:36:08.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5766000000000006e-05 2023-10-20T03:36:08.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.897644+0000", 2023-10-20T03:36:08.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5834000000000001e-05 2023-10-20T03:36:08.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.905485+0000", 2023-10-20T03:36:08.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0078405750000000007 2023-10-20T03:36:08.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.905539+0000", 2023-10-20T03:36:08.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4443e-05 2023-10-20T03:36:08.571 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.571 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.571 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.572 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.bs3 60/56 ECSubWrite(tid=204, reqid=client.4570.0:470, at_version=60'24, trim_to=0'0, roll_forward_to=60'22))", 2023-10-20T03:36:08.572 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.923867+0000", 2023-10-20T03:36:08.572 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.474452083999999, 2023-10-20T03:36:08.572 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014718472999999999, 2023-10-20T03:36:08.573 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.573 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.573 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.574 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.923867+0000", 2023-10-20T03:36:08.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.574 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.574 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.923867+0000", 2023-10-20T03:36:08.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.923868+0000", 2023-10-20T03:36:08.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.063e-06 2023-10-20T03:36:08.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.576 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.576 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924029+0000", 2023-10-20T03:36:08.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000160927 2023-10-20T03:36:08.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924045+0000", 2023-10-20T03:36:08.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5874000000000001e-05 2023-10-20T03:36:08.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.578 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.578 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924054+0000", 2023-10-20T03:36:08.578 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7320000000000004e-06 2023-10-20T03:36:08.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.579 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.579 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.579 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924173+0000", 2023-10-20T03:36:08.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000118713 2023-10-20T03:36:08.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.580 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.580 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.580 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924190+0000", 2023-10-20T03:36:08.580 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7280000000000001e-05 2023-10-20T03:36:08.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.938488+0000", 2023-10-20T03:36:08.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014298132 2023-10-20T03:36:08.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.938586+0000", 2023-10-20T03:36:08.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7751999999999995e-05 2023-10-20T03:36:08.583 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.583 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.583 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.583 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.583 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s3 70/56 ECSubWrite(tid=55, reqid=client.4570.0:556, at_version=70'40, trim_to=0'0, roll_forward_to=60'38))", 2023-10-20T03:36:08.584 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:51.966076+0000", 2023-10-20T03:36:08.584 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 16.432243410000002, 2023-10-20T03:36:08.584 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0089416370000000005, 2023-10-20T03:36:08.584 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.584 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.585 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.585 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.585 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.585 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.966076+0000", 2023-10-20T03:36:08.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.586 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.586 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.586 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.966076+0000", 2023-10-20T03:36:08.586 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.587 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.587 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.587 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.966077+0000", 2023-10-20T03:36:08.587 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6899999999999998e-07 2023-10-20T03:36:08.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.966521+0000", 2023-10-20T03:36:08.588 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00044419099999999998 2023-10-20T03:36:08.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.589 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.589 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.966547+0000", 2023-10-20T03:36:08.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6398999999999999e-05 2023-10-20T03:36:08.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.966572+0000", 2023-10-20T03:36:08.590 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4510000000000001e-05 2023-10-20T03:36:08.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.591 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.591 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.966705+0000", 2023-10-20T03:36:08.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000133022 2023-10-20T03:36:08.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.592 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.592 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.966724+0000", 2023-10-20T03:36:08.592 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9264999999999999e-05 2023-10-20T03:36:08.592 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.592 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.592 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.593 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.974974+0000", 2023-10-20T03:36:08.593 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0082500639999999997 2023-10-20T03:36:08.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.593 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.594 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.975018+0000", 2023-10-20T03:36:08.594 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3717000000000002e-05 2023-10-20T03:36:08.594 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.594 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.594 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.595 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.595 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s3 70/56 ECSubWrite(tid=57, reqid=client.4570.0:558, at_version=70'42, trim_to=0'0, roll_forward_to=70'40))", 2023-10-20T03:36:08.595 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:51.998575+0000", 2023-10-20T03:36:08.597 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 16.399744593000001, 2023-10-20T03:36:08.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013366552, 2023-10-20T03:36:08.597 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.598 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.598 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.598 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.598 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.598 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998575+0000", 2023-10-20T03:36:08.599 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.599 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.599 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.599 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998575+0000", 2023-10-20T03:36:08.600 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.600 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.600 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998577+0000", 2023-10-20T03:36:08.600 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.967e-06 2023-10-20T03:36:08.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.601 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.601 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.601 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998782+0000", 2023-10-20T03:36:08.601 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020551 2023-10-20T03:36:08.602 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.602 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.602 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.602 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998818+0000", 2023-10-20T03:36:08.602 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5879e-05 2023-10-20T03:36:08.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.603 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.603 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998836+0000", 2023-10-20T03:36:08.603 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7583000000000001e-05 2023-10-20T03:36:08.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.604 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.604 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.604 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998975+0000", 2023-10-20T03:36:08.604 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000139231 2023-10-20T03:36:08.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.605 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.605 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.605 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998994+0000", 2023-10-20T03:36:08.605 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8936e-05 2023-10-20T03:36:08.605 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.606 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.606 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.011876+0000", 2023-10-20T03:36:08.606 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012881951000000001 2023-10-20T03:36:08.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.011941+0000", 2023-10-20T03:36:08.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5494999999999995e-05 2023-10-20T03:36:08.607 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.608 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.608 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.608 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.bs3 70/56 ECSubWrite(tid=260, reqid=client.4570.0:561, at_version=70'36, trim_to=0'0, roll_forward_to=61'34))", 2023-10-20T03:36:08.609 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:52.022553+0000", 2023-10-20T03:36:08.609 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 16.375766922, 2023-10-20T03:36:08.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012554222, 2023-10-20T03:36:08.609 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.609 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.610 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.610 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.610 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.610 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.022553+0000", 2023-10-20T03:36:08.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.611 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.611 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.022553+0000", 2023-10-20T03:36:08.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.612 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.612 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.612 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.612 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.022553+0000", 2023-10-20T03:36:08.612 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8000000000000001e-07 2023-10-20T03:36:08.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.613 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.613 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.022966+0000", 2023-10-20T03:36:08.613 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00041324899999999997 2023-10-20T03:36:08.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.022988+0000", 2023-10-20T03:36:08.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1557999999999999e-05 2023-10-20T03:36:08.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.615 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.023003+0000", 2023-10-20T03:36:08.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5150000000000001e-05 2023-10-20T03:36:08.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.023112+0000", 2023-10-20T03:36:08.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000108798 2023-10-20T03:36:08.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.617 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.617 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.023128+0000", 2023-10-20T03:36:08.617 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5977999999999998e-05 2023-10-20T03:36:08.617 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.617 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.618 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.618 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.035026+0000", 2023-10-20T03:36:08.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011898633 2023-10-20T03:36:08.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.035107+0000", 2023-10-20T03:36:08.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0476000000000007e-05 2023-10-20T03:36:08.619 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.619 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.620 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.620 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.620 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s3 71/56 ECSubWrite(tid=63, reqid=client.4570.0:575, at_version=71'46, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:08.620 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:52.983574+0000", 2023-10-20T03:36:08.621 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 15.414745848999999, 2023-10-20T03:36:08.621 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011610976, 2023-10-20T03:36:08.621 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.621 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.621 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.621 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.622 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.622 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.983574+0000", 2023-10-20T03:36:08.622 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.623 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.623 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.983574+0000", 2023-10-20T03:36:08.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.623 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.624 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.983575+0000", 2023-10-20T03:36:08.624 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1429999999999999e-06 2023-10-20T03:36:08.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.624 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984011+0000", 2023-10-20T03:36:08.625 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00043591300000000001 2023-10-20T03:36:08.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.625 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.626 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984039+0000", 2023-10-20T03:36:08.626 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8119999999999998e-05 2023-10-20T03:36:08.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.626 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.627 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984052+0000", 2023-10-20T03:36:08.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3332e-05 2023-10-20T03:36:08.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.627 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984151+0000", 2023-10-20T03:36:08.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8503000000000002e-05 2023-10-20T03:36:08.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984168+0000", 2023-10-20T03:36:08.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7053000000000001e-05 2023-10-20T03:36:08.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.995118+0000", 2023-10-20T03:36:08.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010950616 2023-10-20T03:36:08.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.630 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.630 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.995185+0000", 2023-10-20T03:36:08.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6296000000000004e-05 2023-10-20T03:36:08.631 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.631 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.631 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.632 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s2 74/31 ECSubWrite(tid=65, reqid=client.4570.0:615, at_version=74'48, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:08.632 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:56.030516+0000", 2023-10-20T03:36:08.632 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 12.367803079, 2023-10-20T03:36:08.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010678188, 2023-10-20T03:36:08.633 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:08.633 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:08.633 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:08.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.633 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:08.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030516+0000", 2023-10-20T03:36:08.634 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.634 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:08.634 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030516+0000", 2023-10-20T03:36:08.635 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:08.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.635 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.635 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:08.635 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030517+0000", 2023-10-20T03:36:08.636 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4799999999999997e-07 2023-10-20T03:36:08.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.636 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.636 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:08.636 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030941+0000", 2023-10-20T03:36:08.636 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00042401400000000001 2023-10-20T03:36:08.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.637 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:08.637 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030956+0000", 2023-10-20T03:36:08.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4381e-05 2023-10-20T03:36:08.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:08.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030963+0000", 2023-10-20T03:36:08.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3499999999999999e-06 2023-10-20T03:36:08.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.639 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:08.639 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.031035+0000", 2023-10-20T03:36:08.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1512000000000002e-05 2023-10-20T03:36:08.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.640 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:08.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.031048+0000", 2023-10-20T03:36:08.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3118e-05 2023-10-20T03:36:08.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:08.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.041148+0000", 2023-10-20T03:36:08.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010100029 2023-10-20T03:36:08.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:08.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:08.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:08.642 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.041195+0000", 2023-10-20T03:36:08.642 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6935999999999999e-05 2023-10-20T03:36:08.642 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.643 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.643 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.643 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:08.643 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:08.643 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:08.645 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-10-20T03:36:08.646 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:08.645+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:08.745 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:08.743+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:08.821 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:08.821 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:36:08.821 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:36:08.822 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:08.827 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:36:08.835 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-10-20T03:36:08.846 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:08.841+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:08.947 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:08.944+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:08.978 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:36:08.978 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:36:08.998 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:36:08.999 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:36:08.999 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:08.999 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:36:09.000 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:36:09.000 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:36:09.000 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:09.015 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-10-20T03:36:09.017 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:36:09.017 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:36:09.038 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:36:09.038 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:36:09.047 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:09.044+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:09.054 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:36:09.055 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:36:09.068 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2023-10-20T03:36:09.077 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:36:09.078 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:36:09.078 INFO:tasks.rados.rados.0.smithi170.stdout:296: done (2 left) 2023-10-20T03:36:09.079 INFO:tasks.rados.rados.0.smithi170.stdout:297: read oid 4 snap -1 2023-10-20T03:36:09.079 INFO:tasks.rados.rados.0.smithi170.stdout:297: expect (ObjNum 103 snap 13 seq_num 103) 2023-10-20T03:36:09.079 INFO:tasks.rados.rados.0.smithi170.stdout:298: rmattr oid 39 current snap is 26 2023-10-20T03:36:09.079 INFO:tasks.rados.rados.0.smithi170.stdout:289: expect (ObjNum 122 snap 25 seq_num 122) 2023-10-20T03:36:09.080 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.6 2023-10-20T03:36:09.112 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:36:09.113 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:36:09.129 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:36:09.130 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:36:09.146 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:09.147 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:09.148 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:09.145+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:09.149 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:36:09.150 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-10-20T03:36:09.161 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-10-20T03:36:09.171 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:36:09.171 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:36:09.191 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:36:09.191 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:36:09.207 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:36:09.207 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:36:09.227 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:36:09.228 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:36:09.247 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:36:09.248 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-10-20T03:36:09.249 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:09.247+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:09.268 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:36:09.268 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:36:09.273 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:36:09.274 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:36:09.299 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:09.300 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:09.314 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-10-20T03:36:09.321 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:09.319+0000 7f7215348640 -1 osd.6 28 log_to_monitors true 2023-10-20T03:36:09.350 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:36:09.347+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:36:09.420 INFO:tasks.rados.rados.0.smithi170.stdout:295: finishing write tid 1 to smithi17039300-40 2023-10-20T03:36:09.421 INFO:tasks.rados.rados.0.smithi170.stdout:295: finishing write tid 2 to smithi17039300-40 2023-10-20T03:36:09.421 INFO:tasks.rados.rados.0.smithi170.stdout:295: finishing write tid 3 to smithi17039300-40 2023-10-20T03:36:09.421 INFO:tasks.rados.rados.0.smithi170.stdout:295: finishing write tid 4 to smithi17039300-40 2023-10-20T03:36:09.421 INFO:tasks.rados.rados.0.smithi170.stdout:295: finishing write tid 5 to smithi17039300-40 2023-10-20T03:36:09.421 INFO:tasks.rados.rados.0.smithi170.stdout:295: finishing write tid 6 to smithi17039300-40 2023-10-20T03:36:09.421 INFO:tasks.rados.rados.0.smithi170.stdout:295: oid 40 updating version 0 to 39 2023-10-20T03:36:09.422 INFO:tasks.rados.rados.0.smithi170.stdout:295: oid 40 version 39 is already newer than 38 2023-10-20T03:36:09.422 INFO:tasks.rados.rados.0.smithi170.stdout:295: oid 40 updating version 39 to 40 2023-10-20T03:36:09.422 INFO:tasks.rados.rados.0.smithi170.stdout:295: oid 40 updating version 40 to 42 2023-10-20T03:36:09.422 INFO:tasks.rados.rados.0.smithi170.stdout:295: oid 40 version 42 is already newer than 41 2023-10-20T03:36:09.423 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 40 v 42 (ObjNum 127 snap 26 seq_num 127) dirty exists 2023-10-20T03:36:09.423 INFO:tasks.rados.rados.0.smithi170.stdout:295: left oid 40 (ObjNum 127 snap 26 seq_num 127) 2023-10-20T03:36:09.425 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 63 (ObjNum 129 snap 0 seq_num 3159950816) dirty exists 2023-10-20T03:36:09.425 INFO:tasks.rados.rados.0.smithi170.stdout:297: expect (ObjNum 103 snap 13 seq_num 103) 2023-10-20T03:36:09.451 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:09.448+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:09.475 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:09.476 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:09.490 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-10-20T03:36:09.552 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:36:09.548+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:36:09.554 INFO:tasks.rados.rados.0.smithi170.stdout:289: done (3 left) 2023-10-20T03:36:09.554 INFO:tasks.rados.rados.0.smithi170.stdout:295: done (2 left) 2023-10-20T03:36:09.554 INFO:tasks.rados.rados.0.smithi170.stdout:297: done (1 left) 2023-10-20T03:36:09.555 INFO:tasks.rados.rados.0.smithi170.stdout:298: done (0 left) 2023-10-20T03:36:09.555 INFO:tasks.rados.rados.0.smithi170.stdout:299: rollback oid 10 current snap is 26 2023-10-20T03:36:09.555 INFO:tasks.rados.rados.0.smithi170.stdout:299: done (0 left) 2023-10-20T03:36:09.555 INFO:tasks.rados.rados.0.smithi170.stdout:300: read oid 42 snap -1 2023-10-20T03:36:09.555 INFO:tasks.rados.rados.0.smithi170.stdout:300: expect (ObjNum 60 snap 2 seq_num 60) 2023-10-20T03:36:09.556 INFO:tasks.rados.rados.0.smithi170.stdout:301: delete oid 40 current snap is 26 2023-10-20T03:36:09.560 INFO:tasks.rados.rados.0.smithi170.stdout:301: done (1 left) 2023-10-20T03:36:09.561 INFO:tasks.rados.rados.0.smithi170.stdout:302: rollback oid 7 current snap is 26 2023-10-20T03:36:09.561 INFO:tasks.rados.rados.0.smithi170.stdout:302: done (1 left) 2023-10-20T03:36:09.561 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 32 current snap is 26 2023-10-20T03:36:09.561 INFO:tasks.rados.rados.0.smithi170.stdout:303: seq_num 129 ranges {3375104=622592,3997696=655360,4653056=688128,5341184=212992} 2023-10-20T03:36:09.617 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:09.617 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:09.621 INFO:tasks.rados.rados.0.smithi170.stdout:303: writing smithi17039300-32 from 3375104 to 3997696 tid 1 2023-10-20T03:36:09.629 INFO:tasks.rados.rados.0.smithi170.stdout:303: writing smithi17039300-32 from 3997696 to 4653056 tid 2 2023-10-20T03:36:09.632 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-10-20T03:36:09.636 INFO:tasks.rados.rados.0.smithi170.stdout:303: writing smithi17039300-32 from 4653056 to 5341184 tid 3 2023-10-20T03:36:09.638 INFO:tasks.rados.rados.0.smithi170.stdout:303: writing smithi17039300-32 from 5341184 to 5554176 tid 4 2023-10-20T03:36:09.638 INFO:tasks.rados.rados.0.smithi170.stdout:304: read oid 7 snap -1 2023-10-20T03:36:09.639 INFO:tasks.rados.rados.0.smithi170.stdout:304: expect (ObjNum 66 snap 4 seq_num 66) 2023-10-20T03:36:09.639 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 11 current snap is 26 2023-10-20T03:36:09.639 INFO:tasks.rados.rados.0.smithi170.stdout:305: seq_num 130 ranges {4816896=638976,5455872=671744,6127616=802816,6930432=81920} 2023-10-20T03:36:09.654 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:09.649+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:09.678 INFO:tasks.rados.rados.0.smithi170.stdout:305: writing smithi17039300-11 from 4816896 to 5455872 tid 1 2023-10-20T03:36:09.681 INFO:tasks.rados.rados.0.smithi170.stdout:305: writing smithi17039300-11 from 5455872 to 6127616 tid 2 2023-10-20T03:36:09.686 INFO:tasks.rados.rados.0.smithi170.stdout:305: writing smithi17039300-11 from 6127616 to 6930432 tid 3 2023-10-20T03:36:09.687 INFO:tasks.rados.rados.0.smithi170.stdout:305: writing smithi17039300-11 from 6930432 to 7012352 tid 4 2023-10-20T03:36:09.687 INFO:tasks.rados.rados.0.smithi170.stdout:306: snap_create 2023-10-20T03:36:09.688 INFO:tasks.rados.rados.0.smithi170.stdout:300: expect (ObjNum 60 snap 2 seq_num 60) 2023-10-20T03:36:09.754 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:09.752+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:09.771 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:09.771 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:09.786 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-10-20T03:36:09.788 INFO:tasks.rados.rados.0.smithi170.stdout:304: expect (ObjNum 66 snap 4 seq_num 66) 2023-10-20T03:36:09.816 INFO:tasks.rados.rados.0.smithi170.stdout:303: finishing write tid 1 to smithi17039300-32 2023-10-20T03:36:09.816 INFO:tasks.rados.rados.0.smithi170.stdout:303: finishing write tid 2 to smithi17039300-32 2023-10-20T03:36:09.817 INFO:tasks.rados.rados.0.smithi170.stdout:303: finishing write tid 3 to smithi17039300-32 2023-10-20T03:36:09.817 INFO:tasks.rados.rados.0.smithi170.stdout:303: finishing write tid 4 to smithi17039300-32 2023-10-20T03:36:09.817 INFO:tasks.rados.rados.0.smithi170.stdout:303: finishing write tid 5 to smithi17039300-32 2023-10-20T03:36:09.817 INFO:tasks.rados.rados.0.smithi170.stdout:303: finishing write tid 6 to smithi17039300-32 2023-10-20T03:36:09.817 INFO:tasks.rados.rados.0.smithi170.stdout:303: oid 32 updating version 0 to 15 2023-10-20T03:36:09.818 INFO:tasks.rados.rados.0.smithi170.stdout:303: oid 32 updating version 15 to 16 2023-10-20T03:36:09.818 INFO:tasks.rados.rados.0.smithi170.stdout:303: oid 32 version 16 is already newer than 14 2023-10-20T03:36:09.818 INFO:tasks.rados.rados.0.smithi170.stdout:303: oid 32 updating version 16 to 17 2023-10-20T03:36:09.819 INFO:tasks.rados.rados.0.smithi170.stdout:303: oid 32 version 17 is already newer than 13 2023-10-20T03:36:09.819 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 17 (ObjNum 129 snap 26 seq_num 129) dirty exists 2023-10-20T03:36:09.819 INFO:tasks.rados.rados.0.smithi170.stdout:303: left oid 32 (ObjNum 129 snap 26 seq_num 129) 2023-10-20T03:36:09.819 INFO:tasks.rados.rados.0.smithi170.stdout:305: finishing write tid 1 to smithi17039300-11 2023-10-20T03:36:09.819 INFO:tasks.rados.rados.0.smithi170.stdout:305: finishing write tid 2 to smithi17039300-11 2023-10-20T03:36:09.820 INFO:tasks.rados.rados.0.smithi170.stdout:305: finishing write tid 3 to smithi17039300-11 2023-10-20T03:36:09.820 INFO:tasks.rados.rados.0.smithi170.stdout:305: finishing write tid 4 to smithi17039300-11 2023-10-20T03:36:09.820 INFO:tasks.rados.rados.0.smithi170.stdout:305: finishing write tid 5 to smithi17039300-11 2023-10-20T03:36:09.820 INFO:tasks.rados.rados.0.smithi170.stdout:305: finishing write tid 6 to smithi17039300-11 2023-10-20T03:36:09.820 INFO:tasks.rados.rados.0.smithi170.stdout:305: oid 11 updating version 0 to 44 2023-10-20T03:36:09.821 INFO:tasks.rados.rados.0.smithi170.stdout:305: oid 11 version 44 is already newer than 43 2023-10-20T03:36:09.821 INFO:tasks.rados.rados.0.smithi170.stdout:305: oid 11 updating version 44 to 46 2023-10-20T03:36:09.821 INFO:tasks.rados.rados.0.smithi170.stdout:305: oid 11 updating version 46 to 47 2023-10-20T03:36:09.821 INFO:tasks.rados.rados.0.smithi170.stdout:305: oid 11 version 47 is already newer than 45 2023-10-20T03:36:09.821 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 11 v 47 (ObjNum 130 snap 26 seq_num 130) dirty exists 2023-10-20T03:36:09.822 INFO:tasks.rados.rados.0.smithi170.stdout:305: left oid 11 (ObjNum 130 snap 26 seq_num 130) 2023-10-20T03:36:09.878 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:09.876+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:09.955 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:09.955 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:09.957 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:09.952+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:09.969 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-10-20T03:36:10.058 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:10.053+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:10.096 INFO:tasks.rados.rados.0.smithi170.stdout:300: done (4 left) 2023-10-20T03:36:10.096 INFO:tasks.rados.rados.0.smithi170.stdout:303: done (3 left) 2023-10-20T03:36:10.096 INFO:tasks.rados.rados.0.smithi170.stdout:304: done (2 left) 2023-10-20T03:36:10.096 INFO:tasks.rados.rados.0.smithi170.stdout:305: done (1 left) 2023-10-20T03:36:10.097 INFO:tasks.rados.rados.0.smithi170.stdout:306: done (0 left) 2023-10-20T03:36:10.097 INFO:tasks.rados.rados.0.smithi170.stdout:307: delete oid 40 current snap is 27 2023-10-20T03:36:10.099 INFO:tasks.rados.rados.0.smithi170.stdout:307: done (0 left) 2023-10-20T03:36:10.099 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 21 current snap is 27 2023-10-20T03:36:10.099 INFO:tasks.rados.rados.0.smithi170.stdout:308: seq_num 132 ranges {2146304=671744,2818048=573440,3391488=753664,4145152=229376} 2023-10-20T03:36:10.115 INFO:tasks.rados.rados.0.smithi170.stdout:308: writing smithi17039300-21 from 2146304 to 2818048 tid 1 2023-10-20T03:36:10.118 INFO:tasks.rados.rados.0.smithi170.stdout:308: writing smithi17039300-21 from 2818048 to 3391488 tid 2 2023-10-20T03:36:10.122 INFO:tasks.rados.rados.0.smithi170.stdout:308: writing smithi17039300-21 from 3391488 to 4145152 tid 3 2023-10-20T03:36:10.124 INFO:tasks.rados.rados.0.smithi170.stdout:308: writing smithi17039300-21 from 4145152 to 4374528 tid 4 2023-10-20T03:36:10.124 INFO:tasks.rados.rados.0.smithi170.stdout:309: copy_from oid 48 from oid 1 current snap is 27 2023-10-20T03:36:10.124 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 5 current snap is 27 2023-10-20T03:36:10.124 INFO:tasks.rados.rados.0.smithi170.stdout:310: seq_num 133 ranges {3194880=688128,3883008=720896,4603904=475136,5079040=360448} 2023-10-20T03:36:10.139 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:10.139 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:10.145 INFO:tasks.rados.rados.0.smithi170.stdout:310: writing smithi17039300-5 from 3194880 to 3883008 tid 1 2023-10-20T03:36:10.150 INFO:tasks.rados.rados.0.smithi170.stdout:310: writing smithi17039300-5 from 3883008 to 4603904 tid 2 2023-10-20T03:36:10.152 INFO:tasks.rados.rados.0.smithi170.stdout:310: writing smithi17039300-5 from 4603904 to 5079040 tid 3 2023-10-20T03:36:10.155 INFO:tasks.rados.rados.0.smithi170.stdout:310: writing smithi17039300-5 from 5079040 to 5439488 tid 4 2023-10-20T03:36:10.155 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 42 current snap is 27 2023-10-20T03:36:10.156 INFO:tasks.rados.rados.0.smithi170.stdout:309: finishing copy_from to smithi17039300-48 2023-10-20T03:36:10.156 INFO:tasks.rados.rados.0.smithi170.stdout:309: got expected ENOENT (src dne) 2023-10-20T03:36:10.156 INFO:tasks.rados.rados.0.smithi170.stdout:309: finishing copy_from racing read to smithi17039300-48 2023-10-20T03:36:10.157 INFO:tasks.rados.rados.0.smithi170.stdout:311: seq_num 134 ranges {1720320=704512,2424832=475136,2899968=573440,3473408=507904} 2023-10-20T03:36:10.158 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-10-20T03:36:10.159 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:10.157+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:10.170 INFO:tasks.rados.rados.0.smithi170.stdout:311: writing smithi17039300-42 from 1720320 to 2424832 tid 1 2023-10-20T03:36:10.173 INFO:tasks.rados.rados.0.smithi170.stdout:311: writing smithi17039300-42 from 2424832 to 2899968 tid 2 2023-10-20T03:36:10.174 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:10.171+0000 7f7209125640 -1 osd.6 88 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-20T03:36:10.176 INFO:tasks.rados.rados.0.smithi170.stdout:311: writing smithi17039300-42 from 2899968 to 3473408 tid 3 2023-10-20T03:36:10.181 INFO:tasks.rados.rados.0.smithi170.stdout:311: writing smithi17039300-42 from 3473408 to 3981312 tid 4 2023-10-20T03:36:10.182 INFO:tasks.rados.rados.0.smithi170.stdout:309: done (3 left) 2023-10-20T03:36:10.182 INFO:tasks.rados.rados.0.smithi170.stdout:312: read oid 14 snap 26 2023-10-20T03:36:10.182 INFO:tasks.rados.rados.0.smithi170.stdout:312: expect (ObjNum 99 snap 13 seq_num 99) 2023-10-20T03:36:10.182 INFO:tasks.rados.rados.0.smithi170.stdout:313: delete oid 4 current snap is 27 2023-10-20T03:36:10.183 INFO:tasks.rados.rados.0.smithi170.stdout:308: finishing write tid 1 to smithi17039300-21 2023-10-20T03:36:10.183 INFO:tasks.rados.rados.0.smithi170.stdout:308: finishing write tid 2 to smithi17039300-21 2023-10-20T03:36:10.183 INFO:tasks.rados.rados.0.smithi170.stdout:308: finishing write tid 3 to smithi17039300-21 2023-10-20T03:36:10.186 INFO:tasks.rados.rados.0.smithi170.stdout:308: finishing write tid 4 to smithi17039300-21 2023-10-20T03:36:10.186 INFO:tasks.rados.rados.0.smithi170.stdout:308: finishing write tid 5 to smithi17039300-21 2023-10-20T03:36:10.189 INFO:tasks.rados.rados.0.smithi170.stdout:313: done (4 left) 2023-10-20T03:36:10.189 INFO:tasks.rados.rados.0.smithi170.stdout:314: snap_create 2023-10-20T03:36:10.191 INFO:tasks.rados.rados.0.smithi170.stdout:308: finishing write tid 6 to smithi17039300-21 2023-10-20T03:36:10.192 INFO:tasks.rados.rados.0.smithi170.stdout:308: oid 21 updating version 0 to 68 2023-10-20T03:36:10.192 INFO:tasks.rados.rados.0.smithi170.stdout:308: oid 21 updating version 68 to 69 2023-10-20T03:36:10.192 INFO:tasks.rados.rados.0.smithi170.stdout:308: oid 21 version 69 is already newer than 67 2023-10-20T03:36:10.193 INFO:tasks.rados.rados.0.smithi170.stdout:308: oid 21 version 69 is already newer than 66 2023-10-20T03:36:10.193 INFO:tasks.rados.rados.0.smithi170.stdout:308: oid 21 version 69 is already newer than 65 2023-10-20T03:36:10.193 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 21 v 69 (ObjNum 132 snap 27 seq_num 132) dirty exists 2023-10-20T03:36:10.193 INFO:tasks.rados.rados.0.smithi170.stdout:308: left oid 21 (ObjNum 132 snap 27 seq_num 132) 2023-10-20T03:36:10.195 INFO:tasks.rados.rados.0.smithi170.stdout:310: finishing write tid 1 to smithi17039300-5 2023-10-20T03:36:10.206 INFO:tasks.rados.rados.0.smithi170.stdout:310: finishing write tid 2 to smithi17039300-5 2023-10-20T03:36:10.207 INFO:tasks.rados.rados.0.smithi170.stdout:310: finishing write tid 3 to smithi17039300-5 2023-10-20T03:36:10.210 INFO:tasks.rados.rados.0.smithi170.stdout:310: finishing write tid 4 to smithi17039300-5 2023-10-20T03:36:10.210 INFO:tasks.rados.rados.0.smithi170.stdout:310: finishing write tid 5 to smithi17039300-5 2023-10-20T03:36:10.212 INFO:tasks.rados.rados.0.smithi170.stdout:310: finishing write tid 6 to smithi17039300-5 2023-10-20T03:36:10.213 INFO:tasks.rados.rados.0.smithi170.stdout:310: oid 5 updating version 0 to 46 2023-10-20T03:36:10.213 INFO:tasks.rados.rados.0.smithi170.stdout:310: oid 5 updating version 46 to 50 2023-10-20T03:36:10.213 INFO:tasks.rados.rados.0.smithi170.stdout:310: oid 5 version 50 is already newer than 48 2023-10-20T03:36:10.213 INFO:tasks.rados.rados.0.smithi170.stdout:310: oid 5 version 50 is already newer than 49 2023-10-20T03:36:10.213 INFO:tasks.rados.rados.0.smithi170.stdout:310: oid 5 version 50 is already newer than 47 2023-10-20T03:36:10.214 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 50 (ObjNum 133 snap 27 seq_num 133) dirty exists 2023-10-20T03:36:10.214 INFO:tasks.rados.rados.0.smithi170.stdout:310: left oid 5 (ObjNum 133 snap 27 seq_num 133) 2023-10-20T03:36:10.221 INFO:tasks.rados.rados.0.smithi170.stdout:311: finishing write tid 1 to smithi17039300-42 2023-10-20T03:36:10.221 INFO:tasks.rados.rados.0.smithi170.stdout:311: finishing write tid 2 to smithi17039300-42 2023-10-20T03:36:10.222 INFO:tasks.rados.rados.0.smithi170.stdout:311: finishing write tid 3 to smithi17039300-42 2023-10-20T03:36:10.222 INFO:tasks.rados.rados.0.smithi170.stdout:311: finishing write tid 4 to smithi17039300-42 2023-10-20T03:36:10.222 INFO:tasks.rados.rados.0.smithi170.stdout:311: finishing write tid 5 to smithi17039300-42 2023-10-20T03:36:10.225 INFO:tasks.rados.rados.0.smithi170.stdout:311: finishing write tid 6 to smithi17039300-42 2023-10-20T03:36:10.225 INFO:tasks.rados.rados.0.smithi170.stdout:311: oid 42 updating version 0 to 65 2023-10-20T03:36:10.225 INFO:tasks.rados.rados.0.smithi170.stdout:311: oid 42 updating version 65 to 69 2023-10-20T03:36:10.226 INFO:tasks.rados.rados.0.smithi170.stdout:311: oid 42 version 69 is already newer than 67 2023-10-20T03:36:10.226 INFO:tasks.rados.rados.0.smithi170.stdout:311: oid 42 version 69 is already newer than 66 2023-10-20T03:36:10.226 INFO:tasks.rados.rados.0.smithi170.stdout:311: oid 42 version 69 is already newer than 68 2023-10-20T03:36:10.226 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 42 v 69 (ObjNum 134 snap 27 seq_num 134) dirty exists 2023-10-20T03:36:10.226 INFO:tasks.rados.rados.0.smithi170.stdout:311: left oid 42 (ObjNum 134 snap 27 seq_num 134) 2023-10-20T03:36:10.227 INFO:tasks.rados.rados.0.smithi170.stdout:312: expect (ObjNum 99 snap 13 seq_num 99) 2023-10-20T03:36:10.261 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:10.256+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:10.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-10-20T03:36:10.296 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:10.296 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:10.310 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-10-20T03:36:10.362 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:10.357+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:10.419 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:36:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:36:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2023-10-20T03:36:10.419 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:36:10.436 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:10.436 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:10.451 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-10-20T03:36:10.463 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:10.460+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:10.564 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:10.563+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:10.576 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:10.576 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:10.590 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-10-20T03:36:10.665 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:10.665+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:10.720 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:10.721 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:10.735 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-10-20T03:36:10.766 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:10.761+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:10.856 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:10.856 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:10.867 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:10.865+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:10.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-10-20T03:36:10.968 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:10.964+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:10.989 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:10.989 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:11.003 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-10-20T03:36:11.069 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:11.065+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:11.097 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-10-20T03:36:11.112 INFO:tasks.rados.rados.0.smithi170.stdout:308: done (4 left) 2023-10-20T03:36:11.112 INFO:tasks.rados.rados.0.smithi170.stdout:310: done (3 left) 2023-10-20T03:36:11.113 INFO:tasks.rados.rados.0.smithi170.stdout:311: done (2 left) 2023-10-20T03:36:11.113 INFO:tasks.rados.rados.0.smithi170.stdout:312: done (1 left) 2023-10-20T03:36:11.113 INFO:tasks.rados.rados.0.smithi170.stdout:314: done (0 left) 2023-10-20T03:36:11.113 INFO:tasks.rados.rados.0.smithi170.stdout:315: read oid 3 snap -1 2023-10-20T03:36:11.113 INFO:tasks.rados.rados.0.smithi170.stdout:315: expect (ObjNum 2 snap 0 seq_num 2) 2023-10-20T03:36:11.114 INFO:tasks.rados.rados.0.smithi170.stdout:316: snap_create 2023-10-20T03:36:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:11.131 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:11.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-10-20T03:36:11.170 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:11.167+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:11.261 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:11.262 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:11.271 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:11.268+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:11.277 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:36:11.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-10-20T03:36:11.372 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:36:11.368+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:36:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:11.391 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:11.405 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-10-20T03:36:11.423 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:36:11.423 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.441 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:36:11.441 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.461 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:36:11.461 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.473 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:36:11.471+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:36:11.481 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:36:11.482 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.498 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:36:11.498 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.516 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:36:11.516 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.538 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2023-10-20T03:36:11.538 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.558 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:11.559 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:11.560 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:36:11.560 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-10-20T03:36:11.574 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:11.571+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:11.578 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:36:11.579 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.598 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:36:11.598 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.620 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:36:11.621 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.642 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:36:11.643 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.662 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:36:11.662 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.675 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:11.673+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:11.683 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:11.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-10-20T03:36:11.776 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:11.773+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:11.810 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:11.810 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:11.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-10-20T03:36:11.877 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:11.872+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:11.888 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:36:11.888 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:36:11.889 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:36:11.884+0000 7fe8dbfff640 0 -- 172.21.15.33:0/3671100552 >> [v2:172.21.15.64:6816/2940007286,v1:172.21.15.64:6817/2940007286] conn(0x7fe8dc21f210 msgr2=0x7fe8dc221620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-10-20T03:36:11.961 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:36:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:36:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2023-10-20T03:36:11.961 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:36:11.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-10-20T03:36:11.977 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:11.973+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:12.079 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:12.076+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:12.101 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:36:12.101 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:36:12.101 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2023-10-20T03:36:12.102 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2023-10-20T03:36:12.102 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:36:12.109 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2023-10-20T03:36:12.117 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-10-20T03:36:12.118 INFO:tasks.rados.rados.0.smithi170.stdout:316: done (1 left) 2023-10-20T03:36:12.118 INFO:tasks.rados.rados.0.smithi170.stdout:317: snap_create 2023-10-20T03:36:12.124 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:36:12.125 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:36:12.155 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:36:12.155 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:36:12.179 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:12.177+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:12.246 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:36:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2023-10-20T03:36:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2023-10-20T03:36:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2023-10-20T03:36:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:36:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.329122+0000", 2023-10-20T03:36:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.914806093999999, 2023-10-20T03:36:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034738698999999998, 2023-10-20T03:36:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329122+0000", 2023-10-20T03:36:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329122+0000", 2023-10-20T03:36:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329122+0000", 2023-10-20T03:36:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6399999999999998e-07 2023-10-20T03:36:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329551+0000", 2023-10-20T03:36:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042853900000000002 2023-10-20T03:36:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329577+0000", 2023-10-20T03:36:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6075e-05 2023-10-20T03:36:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329594+0000", 2023-10-20T03:36:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7220000000000001e-05 2023-10-20T03:36:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329788+0000", 2023-10-20T03:36:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000193444 2023-10-20T03:36:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329815+0000", 2023-10-20T03:36:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7107e-05 2023-10-20T03:36:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.363772+0000", 2023-10-20T03:36:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033956919000000002 2023-10-20T03:36:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.363861+0000", 2023-10-20T03:36:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9030999999999995e-05 2023-10-20T03:36:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 43/31 ECSubWrite(tid=81, reqid=client.4570.0:307, at_version=43'15, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:36:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.342678+0000", 2023-10-20T03:36:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.901250011999998, 2023-10-20T03:36:12.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021268994999999999, 2023-10-20T03:36:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.342678+0000", 2023-10-20T03:36:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.342678+0000", 2023-10-20T03:36:12.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.342680+0000", 2023-10-20T03:36:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.736e-06 2023-10-20T03:36:12.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343038+0000", 2023-10-20T03:36:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035823799999999998 2023-10-20T03:36:12.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343066+0000", 2023-10-20T03:36:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7509e-05 2023-10-20T03:36:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343081+0000", 2023-10-20T03:36:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5676e-05 2023-10-20T03:36:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343490+0000", 2023-10-20T03:36:12.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040832599999999998 2023-10-20T03:36:12.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.343517+0000", 2023-10-20T03:36:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7639000000000001e-05 2023-10-20T03:36:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.363880+0000", 2023-10-20T03:36:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020362392999999999 2023-10-20T03:36:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.363947+0000", 2023-10-20T03:36:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7477999999999993e-05 2023-10-20T03:36:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s3 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:36:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.357002+0000", 2023-10-20T03:36:12.270 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.886926268000003, 2023-10-20T03:36:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024440087999999999, 2023-10-20T03:36:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357002+0000", 2023-10-20T03:36:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357002+0000", 2023-10-20T03:36:12.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357004+0000", 2023-10-20T03:36:12.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.883e-06 2023-10-20T03:36:12.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357470+0000", 2023-10-20T03:36:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046657099999999999 2023-10-20T03:36:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357514+0000", 2023-10-20T03:36:12.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3288999999999998e-05 2023-10-20T03:36:12.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357533+0000", 2023-10-20T03:36:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9123e-05 2023-10-20T03:36:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357687+0000", 2023-10-20T03:36:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000154389 2023-10-20T03:36:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357703+0000", 2023-10-20T03:36:12.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6062999999999999e-05 2023-10-20T03:36:12.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.381375+0000", 2023-10-20T03:36:12.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023672364000000001 2023-10-20T03:36:12.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.381442+0000", 2023-10-20T03:36:12.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6406000000000001e-05 2023-10-20T03:36:12.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.281 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs4 53/47 ECSubWrite(tid=57, reqid=client.4570.0:408, at_version=53'31, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:35.448730+0000", 2023-10-20T03:36:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.795197545999997, 2023-10-20T03:36:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015562098999999999, 2023-10-20T03:36:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.282 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.448730+0000", 2023-10-20T03:36:12.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.448730+0000", 2023-10-20T03:36:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.448732+0000", 2023-10-20T03:36:12.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099999999999999e-06 2023-10-20T03:36:12.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449349+0000", 2023-10-20T03:36:12.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061657400000000005 2023-10-20T03:36:12.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449378+0000", 2023-10-20T03:36:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9374999999999999e-05 2023-10-20T03:36:12.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449401+0000", 2023-10-20T03:36:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.296e-05 2023-10-20T03:36:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449514+0000", 2023-10-20T03:36:12.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113174 2023-10-20T03:36:12.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.449533+0000", 2023-10-20T03:36:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8717000000000001e-05 2023-10-20T03:36:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.464194+0000", 2023-10-20T03:36:12.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014661502 2023-10-20T03:36:12.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.464293+0000", 2023-10-20T03:36:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8287000000000003e-05 2023-10-20T03:36:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs4 53/47 ECSubWrite(tid=58, reqid=client.4570.0:409, at_version=53'32, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:35.472576+0000", 2023-10-20T03:36:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.771351754000001, 2023-10-20T03:36:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022673781, 2023-10-20T03:36:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472576+0000", 2023-10-20T03:36:12.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472576+0000", 2023-10-20T03:36:12.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472578+0000", 2023-10-20T03:36:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7370000000000001e-06 2023-10-20T03:36:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472994+0000", 2023-10-20T03:36:12.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041626100000000002 2023-10-20T03:36:12.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473024+0000", 2023-10-20T03:36:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9278999999999999e-05 2023-10-20T03:36:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473046+0000", 2023-10-20T03:36:12.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2353000000000001e-05 2023-10-20T03:36:12.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473185+0000", 2023-10-20T03:36:12.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013938399999999999 2023-10-20T03:36:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473201+0000", 2023-10-20T03:36:12.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5342999999999999e-05 2023-10-20T03:36:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.495162+0000", 2023-10-20T03:36:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021961633000000001 2023-10-20T03:36:12.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.495250+0000", 2023-10-20T03:36:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7791e-05 2023-10-20T03:36:12.303 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.304 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 54/31 ECSubWrite(tid=65, reqid=client.4570.0:418, at_version=54'32, trim_to=0'0, roll_forward_to=51'31))", 2023-10-20T03:36:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:36.458790+0000", 2023-10-20T03:36:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.785138253, 2023-10-20T03:36:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015152827000000001, 2023-10-20T03:36:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.305 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.458790+0000", 2023-10-20T03:36:12.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.458790+0000", 2023-10-20T03:36:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.458791+0000", 2023-10-20T03:36:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5200000000000001e-06 2023-10-20T03:36:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.459266+0000", 2023-10-20T03:36:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047446599999999999 2023-10-20T03:36:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.459297+0000", 2023-10-20T03:36:12.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0948999999999998e-05 2023-10-20T03:36:12.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.459312+0000", 2023-10-20T03:36:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5526e-05 2023-10-20T03:36:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.459439+0000", 2023-10-20T03:36:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012685800000000001 2023-10-20T03:36:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.459456+0000", 2023-10-20T03:36:12.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7289e-05 2023-10-20T03:36:12.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.473843+0000", 2023-10-20T03:36:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014386421 2023-10-20T03:36:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.473943+0000", 2023-10-20T03:36:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9797999999999996e-05 2023-10-20T03:36:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 60/31 ECSubWrite(tid=190, reqid=client.4570.0:452, at_version=60'54, trim_to=0'0, roll_forward_to=58'48))", 2023-10-20T03:36:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:41.856865+0000", 2023-10-20T03:36:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.387063246, 2023-10-20T03:36:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016682313000000001, 2023-10-20T03:36:12.316 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.856865+0000", 2023-10-20T03:36:12.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.856865+0000", 2023-10-20T03:36:12.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.856866+0000", 2023-10-20T03:36:12.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.09e-07 2023-10-20T03:36:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857372+0000", 2023-10-20T03:36:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050624299999999997 2023-10-20T03:36:12.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857398+0000", 2023-10-20T03:36:12.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5803e-05 2023-10-20T03:36:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857415+0000", 2023-10-20T03:36:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7546000000000001e-05 2023-10-20T03:36:12.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857572+0000", 2023-10-20T03:36:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015650700000000001 2023-10-20T03:36:12.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857601+0000", 2023-10-20T03:36:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9513e-05 2023-10-20T03:36:12.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.873347+0000", 2023-10-20T03:36:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015746229 2023-10-20T03:36:12.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.873547+0000", 2023-10-20T03:36:12.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019966299999999999 2023-10-20T03:36:12.326 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 60/31 ECSubWrite(tid=45, reqid=osd.7.0:45, at_version=60'32, trim_to=0'0, roll_forward_to=56'30))", 2023-10-20T03:36:12.327 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:41.857583+0000", 2023-10-20T03:36:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.386345452, 2023-10-20T03:36:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016145725999999999, 2023-10-20T03:36:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.328 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857583+0000", 2023-10-20T03:36:12.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857583+0000", 2023-10-20T03:36:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857583+0000", 2023-10-20T03:36:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0699999999999996e-07 2023-10-20T03:36:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857634+0000", 2023-10-20T03:36:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0785999999999998e-05 2023-10-20T03:36:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857660+0000", 2023-10-20T03:36:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6363000000000002e-05 2023-10-20T03:36:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857682+0000", 2023-10-20T03:36:12.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1559000000000001e-05 2023-10-20T03:36:12.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857833+0000", 2023-10-20T03:36:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015113 2023-10-20T03:36:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.857849+0000", 2023-10-20T03:36:12.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5906999999999999e-05 2023-10-20T03:36:12.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.873555+0000", 2023-10-20T03:36:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015706210000000002 2023-10-20T03:36:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.873728+0000", 2023-10-20T03:36:12.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017306400000000001 2023-10-20T03:36:12.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.339 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 60/31 ECSubWrite(tid=46, reqid=osd.7.0:46, at_version=60'34, trim_to=0'0, roll_forward_to=56'30))", 2023-10-20T03:36:12.339 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:41.863005+0000", 2023-10-20T03:36:12.339 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.380923242000001, 2023-10-20T03:36:12.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034635154000000001, 2023-10-20T03:36:12.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.863005+0000", 2023-10-20T03:36:12.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.863005+0000", 2023-10-20T03:36:12.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.863006+0000", 2023-10-20T03:36:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2023-10-20T03:36:12.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.863036+0000", 2023-10-20T03:36:12.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0494e-05 2023-10-20T03:36:12.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.863048+0000", 2023-10-20T03:36:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1643e-05 2023-10-20T03:36:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.863056+0000", 2023-10-20T03:36:12.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7190000000000007e-06 2023-10-20T03:36:12.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.870966+0000", 2023-10-20T03:36:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079100249999999993 2023-10-20T03:36:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.870994+0000", 2023-10-20T03:36:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8007000000000001e-05 2023-10-20T03:36:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.897561+0000", 2023-10-20T03:36:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026566751999999999 2023-10-20T03:36:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.897640+0000", 2023-10-20T03:36:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8769000000000004e-05 2023-10-20T03:36:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 60/31 ECSubWrite(tid=195, reqid=client.4570.0:453, at_version=60'55, trim_to=0'0, roll_forward_to=60'52))", 2023-10-20T03:36:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:41.877092+0000", 2023-10-20T03:36:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.366836428999999, 2023-10-20T03:36:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028928008000000002, 2023-10-20T03:36:12.350 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877092+0000", 2023-10-20T03:36:12.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877092+0000", 2023-10-20T03:36:12.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877093+0000", 2023-10-20T03:36:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.657e-06 2023-10-20T03:36:12.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877617+0000", 2023-10-20T03:36:12.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052373199999999995 2023-10-20T03:36:12.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877655+0000", 2023-10-20T03:36:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8253000000000001e-05 2023-10-20T03:36:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877685+0000", 2023-10-20T03:36:12.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0096000000000001e-05 2023-10-20T03:36:12.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877884+0000", 2023-10-20T03:36:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000198686 2023-10-20T03:36:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.877915+0000", 2023-10-20T03:36:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0809000000000003e-05 2023-10-20T03:36:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.905863+0000", 2023-10-20T03:36:12.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027948218 2023-10-20T03:36:12.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:41.906020+0000", 2023-10-20T03:36:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015655700000000001 2023-10-20T03:36:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 64/31 ECSubWrite(tid=246, reqid=client.4570.0:539, at_version=64'63, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:36:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.897808+0000", 2023-10-20T03:36:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.346119746999999, 2023-10-20T03:36:12.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031221839000000001, 2023-10-20T03:36:12.362 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.362 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.362 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.897808+0000", 2023-10-20T03:36:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.897808+0000", 2023-10-20T03:36:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.897809+0000", 2023-10-20T03:36:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5999999999999999e-07 2023-10-20T03:36:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898180+0000", 2023-10-20T03:36:12.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000371443 2023-10-20T03:36:12.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898207+0000", 2023-10-20T03:36:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7059e-05 2023-10-20T03:36:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898225+0000", 2023-10-20T03:36:12.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7416999999999999e-05 2023-10-20T03:36:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898433+0000", 2023-10-20T03:36:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020842899999999999 2023-10-20T03:36:12.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898469+0000", 2023-10-20T03:36:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5896000000000003e-05 2023-10-20T03:36:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.928921+0000", 2023-10-20T03:36:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030451733000000002 2023-10-20T03:36:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.929030+0000", 2023-10-20T03:36:12.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109502 2023-10-20T03:36:12.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.371 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 64/31 ECSubWrite(tid=247, reqid=client.4570.0:540, at_version=64'64, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:36:12.372 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.912251+0000", 2023-10-20T03:36:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.331677331000002, 2023-10-20T03:36:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029618076, 2023-10-20T03:36:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912251+0000", 2023-10-20T03:36:12.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912251+0000", 2023-10-20T03:36:12.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912252+0000", 2023-10-20T03:36:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7099999999999999e-06 2023-10-20T03:36:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912538+0000", 2023-10-20T03:36:12.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028583800000000001 2023-10-20T03:36:12.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912591+0000", 2023-10-20T03:36:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2695999999999998e-05 2023-10-20T03:36:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.912612+0000", 2023-10-20T03:36:12.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0556000000000001e-05 2023-10-20T03:36:12.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.917025+0000", 2023-10-20T03:36:12.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0044131070000000003 2023-10-20T03:36:12.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.917041+0000", 2023-10-20T03:36:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.63e-05 2023-10-20T03:36:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.941770+0000", 2023-10-20T03:36:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024729004999999998 2023-10-20T03:36:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.941869+0000", 2023-10-20T03:36:12.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8863999999999994e-05 2023-10-20T03:36:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s6 64/31 ECSubWrite(tid=248, reqid=client.4570.0:541, at_version=64'65, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:36:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.926506+0000", 2023-10-20T03:36:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.317421745000001, 2023-10-20T03:36:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016346448999999999, 2023-10-20T03:36:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.384 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926506+0000", 2023-10-20T03:36:12.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926506+0000", 2023-10-20T03:36:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926508+0000", 2023-10-20T03:36:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7239999999999999e-06 2023-10-20T03:36:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926740+0000", 2023-10-20T03:36:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023180499999999999 2023-10-20T03:36:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926772+0000", 2023-10-20T03:36:12.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1803e-05 2023-10-20T03:36:12.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926795+0000", 2023-10-20T03:36:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3105999999999999e-05 2023-10-20T03:36:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.929124+0000", 2023-10-20T03:36:12.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002329607 2023-10-20T03:36:12.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.929158+0000", 2023-10-20T03:36:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3949000000000003e-05 2023-10-20T03:36:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.942766+0000", 2023-10-20T03:36:12.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013607842 2023-10-20T03:36:12.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.942853+0000", 2023-10-20T03:36:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6613000000000006e-05 2023-10-20T03:36:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 69/31 ECSubWrite(tid=95, reqid=client.4570.0:552, at_version=69'41, trim_to=0'0, roll_forward_to=56'39))", 2023-10-20T03:36:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:50.953519+0000", 2023-10-20T03:36:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.290409434000001, 2023-10-20T03:36:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032960562999999998, 2023-10-20T03:36:12.395 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.953519+0000", 2023-10-20T03:36:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.953519+0000", 2023-10-20T03:36:12.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.953519+0000", 2023-10-20T03:36:12.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7e-07 2023-10-20T03:36:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.954577+0000", 2023-10-20T03:36:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001058072 2023-10-20T03:36:12.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.954609+0000", 2023-10-20T03:36:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1637000000000002e-05 2023-10-20T03:36:12.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.954630+0000", 2023-10-20T03:36:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1719e-05 2023-10-20T03:36:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.954811+0000", 2023-10-20T03:36:12.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018069399999999999 2023-10-20T03:36:12.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.954842+0000", 2023-10-20T03:36:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1053000000000002e-05 2023-10-20T03:36:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.986384+0000", 2023-10-20T03:36:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031542002999999999 2023-10-20T03:36:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:50.986479+0000", 2023-10-20T03:36:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5014999999999997e-05 2023-10-20T03:36:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.406 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.3s5 70/56 ECSubWrite(tid=55, reqid=client.4570.0:556, at_version=70'40, trim_to=0'0, roll_forward_to=60'38))", 2023-10-20T03:36:12.406 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.966477+0000", 2023-10-20T03:36:12.406 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.277451041999999, 2023-10-20T03:36:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016354670000000002, 2023-10-20T03:36:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966477+0000", 2023-10-20T03:36:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966477+0000", 2023-10-20T03:36:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966495+0000", 2023-10-20T03:36:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7666e-05 2023-10-20T03:36:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966636+0000", 2023-10-20T03:36:12.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141755 2023-10-20T03:36:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966667+0000", 2023-10-20T03:36:12.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0846000000000002e-05 2023-10-20T03:36:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966684+0000", 2023-10-20T03:36:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7173e-05 2023-10-20T03:36:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967319+0000", 2023-10-20T03:36:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063411999999999995 2023-10-20T03:36:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967349+0000", 2023-10-20T03:36:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0295999999999999e-05 2023-10-20T03:36:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.982759+0000", 2023-10-20T03:36:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015409707999999999 2023-10-20T03:36:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.982832+0000", 2023-10-20T03:36:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3106000000000001e-05 2023-10-20T03:36:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 70/31 ECSubWrite(tid=48, reqid=client.4570.0:567, at_version=70'36, trim_to=0'0, roll_forward_to=60'34))", 2023-10-20T03:36:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.044845+0000", 2023-10-20T03:36:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.199083121000001, 2023-10-20T03:36:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018144609999999999, 2023-10-20T03:36:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.044845+0000", 2023-10-20T03:36:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.044845+0000", 2023-10-20T03:36:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.044846+0000", 2023-10-20T03:36:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3300000000000001e-07 2023-10-20T03:36:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.045358+0000", 2023-10-20T03:36:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00051205900000000004 2023-10-20T03:36:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.045396+0000", 2023-10-20T03:36:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8729000000000001e-05 2023-10-20T03:36:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.045422+0000", 2023-10-20T03:36:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5256e-05 2023-10-20T03:36:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.045580+0000", 2023-10-20T03:36:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015875899999999999 2023-10-20T03:36:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.045604+0000", 2023-10-20T03:36:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3941e-05 2023-10-20T03:36:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.062900+0000", 2023-10-20T03:36:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017295535000000001 2023-10-20T03:36:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.062990+0000", 2023-10-20T03:36:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9598000000000005e-05 2023-10-20T03:36:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.9s1 70/31 ECSubWrite(tid=49, reqid=client.4570.0:568, at_version=70'37, trim_to=0'0, roll_forward_to=60'34))", 2023-10-20T03:36:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.065413+0000", 2023-10-20T03:36:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.178515134000001, 2023-10-20T03:36:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015280958000000001, 2023-10-20T03:36:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065413+0000", 2023-10-20T03:36:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065413+0000", 2023-10-20T03:36:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065414+0000", 2023-10-20T03:36:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.198e-06 2023-10-20T03:36:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065685+0000", 2023-10-20T03:36:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000271153 2023-10-20T03:36:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065709+0000", 2023-10-20T03:36:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3431999999999999e-05 2023-10-20T03:36:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065727+0000", 2023-10-20T03:36:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8343999999999999e-05 2023-10-20T03:36:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065828+0000", 2023-10-20T03:36:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100807 2023-10-20T03:36:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.065842+0000", 2023-10-20T03:36:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4205999999999999e-05 2023-10-20T03:36:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.080645+0000", 2023-10-20T03:36:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014803278 2023-10-20T03:36:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.080694+0000", 2023-10-20T03:36:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8539999999999999e-05 2023-10-20T03:36:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.3s5 71/56 ECSubWrite(tid=63, reqid=client.4570.0:575, at_version=71'46, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.983660+0000", 2023-10-20T03:36:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.260267669000001, 2023-10-20T03:36:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022066749, 2023-10-20T03:36:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.983660+0000", 2023-10-20T03:36:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.983660+0000", 2023-10-20T03:36:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.983677+0000", 2023-10-20T03:36:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7067000000000001e-05 2023-10-20T03:36:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984054+0000", 2023-10-20T03:36:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037618299999999999 2023-10-20T03:36:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984157+0000", 2023-10-20T03:36:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103799 2023-10-20T03:36:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984180+0000", 2023-10-20T03:36:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2178999999999999e-05 2023-10-20T03:36:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984376+0000", 2023-10-20T03:36:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195937 2023-10-20T03:36:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984413+0000", 2023-10-20T03:36:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7076000000000002e-05 2023-10-20T03:36:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.005641+0000", 2023-10-20T03:36:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021228526000000001 2023-10-20T03:36:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.005727+0000", 2023-10-20T03:36:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5982000000000004e-05 2023-10-20T03:36:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.3s5 71/56 ECSubWrite(tid=64, reqid=client.4570.0:576, at_version=71'47, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:53.006396+0000", 2023-10-20T03:36:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.237532380000001, 2023-10-20T03:36:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027149204999999999, 2023-10-20T03:36:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006396+0000", 2023-10-20T03:36:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006396+0000", 2023-10-20T03:36:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006398+0000", 2023-10-20T03:36:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1959999999999998e-06 2023-10-20T03:36:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006808+0000", 2023-10-20T03:36:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000409694 2023-10-20T03:36:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006863+0000", 2023-10-20T03:36:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5439000000000002e-05 2023-10-20T03:36:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006895+0000", 2023-10-20T03:36:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1715999999999999e-05 2023-10-20T03:36:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007082+0000", 2023-10-20T03:36:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018744399999999999 2023-10-20T03:36:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007118+0000", 2023-10-20T03:36:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5373000000000002e-05 2023-10-20T03:36:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.033480+0000", 2023-10-20T03:36:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026362421 2023-10-20T03:36:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.033545+0000", 2023-10-20T03:36:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4921999999999999e-05 2023-10-20T03:36:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.0s3 75/31 ECSubWrite(tid=97, reqid=client.4570.0:628, at_version=75'43, trim_to=0'0, roll_forward_to=69'41))", 2023-10-20T03:36:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:57.035594+0000", 2023-10-20T03:36:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.208334202, 2023-10-20T03:36:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027768920999999998, 2023-10-20T03:36:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.035594+0000", 2023-10-20T03:36:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.035594+0000", 2023-10-20T03:36:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.035594+0000", 2023-10-20T03:36:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-07 2023-10-20T03:36:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.036737+0000", 2023-10-20T03:36:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001142997 2023-10-20T03:36:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.036754+0000", 2023-10-20T03:36:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6359999999999999e-05 2023-10-20T03:36:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.036762+0000", 2023-10-20T03:36:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3529999999999995e-06 2023-10-20T03:36:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.036853+0000", 2023-10-20T03:36:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1187999999999995e-05 2023-10-20T03:36:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.036863+0000", 2023-10-20T03:36:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1770000000000004e-06 2023-10-20T03:36:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.063311+0000", 2023-10-20T03:36:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02644821 2023-10-20T03:36:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.063363+0000", 2023-10-20T03:36:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1993000000000002e-05 2023-10-20T03:36:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.473 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:36:12.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-10-20T03:36:12.476 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:12.471+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:12.576 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:12.573+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:12.650 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:36:12.651 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:36:12.651 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2023-10-20T03:36:12.651 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:36:12.665 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-10-20T03:36:12.677 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:12.676+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:12.778 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:36:12.775+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:36:12.793 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:36:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:36:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2023-10-20T03:36:12.793 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2023-10-20T03:36:12.794 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:36:12.807 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-10-20T03:36:12.879 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:12.876+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:12.976 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:36:12.976 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2023-10-20T03:36:12.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2023-10-20T03:36:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2023-10-20T03:36:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:36:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.330272+0000", 2023-10-20T03:36:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.644313058999998, 2023-10-20T03:36:12.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035047879999999997, 2023-10-20T03:36:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330272+0000", 2023-10-20T03:36:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330272+0000", 2023-10-20T03:36:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330273+0000", 2023-10-20T03:36:12.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6300000000000005e-07 2023-10-20T03:36:12.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330709+0000", 2023-10-20T03:36:12.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043642900000000001 2023-10-20T03:36:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330745+0000", 2023-10-20T03:36:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6161999999999997e-05 2023-10-20T03:36:12.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330770+0000", 2023-10-20T03:36:12.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5151999999999999e-05 2023-10-20T03:36:12.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.331008+0000", 2023-10-20T03:36:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023709800000000001 2023-10-20T03:36:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:12.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.331035+0000", 2023-10-20T03:36:12.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7178999999999998e-05 2023-10-20T03:36:12.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:12.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365221+0000", 2023-10-20T03:36:12.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034186753 2023-10-20T03:36:12.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.365320+0000", 2023-10-20T03:36:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8543999999999997e-05 2023-10-20T03:36:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:12.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:423 3.3s0 3:c87f890b:::smithi17039300-50:head [append 0~786432] snapc f=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-10-20T03:36:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:36.491346+0000", 2023-10-20T03:36:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.483238843000002, 2023-10-20T03:36:12.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.014221971, 2023-10-20T03:36:12.989 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:12.989 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:12.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:12.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:12.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 423 2023-10-20T03:36:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:12.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.491346+0000", 2023-10-20T03:36:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.491346+0000", 2023-10-20T03:36:12.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.491349+0000", 2023-10-20T03:36:12.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7800000000000001e-06 2023-10-20T03:36:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.492722+0000", 2023-10-20T03:36:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013730470000000001 2023-10-20T03:36:12.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.492757+0000", 2023-10-20T03:36:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4798999999999997e-05 2023-10-20T03:36:12.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.492773+0000", 2023-10-20T03:36:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6192999999999999e-05 2023-10-20T03:36:12.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.492921+0000", 2023-10-20T03:36:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014757400000000001 2023-10-20T03:36:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2023-10-20T03:36:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:36.492926+0000", 2023-10-20T03:36:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5430000000000001e-06 2023-10-20T03:36:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.460373+0000", 2023-10-20T03:36:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.967446264 2023-10-20T03:36:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.461124+0000", 2023-10-20T03:36:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075168299999999995 2023-10-20T03:36:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:12.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.465469+0000", 2023-10-20T03:36:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043443919999999999 2023-10-20T03:36:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.485210+0000", 2023-10-20T03:36:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019740981000000001 2023-10-20T03:36:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.504890+0000", 2023-10-20T03:36:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019680842 2023-10-20T03:36:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.505568+0000", 2023-10-20T03:36:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067787299999999995 2023-10-20T03:36:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 64/31 ECSubWrite(tid=246, reqid=client.4570.0:539, at_version=64'63, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:36:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.898251+0000", 2023-10-20T03:36:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.076334417000002, 2023-10-20T03:36:13.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028526125999999999, 2023-10-20T03:36:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898251+0000", 2023-10-20T03:36:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898251+0000", 2023-10-20T03:36:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898251+0000", 2023-10-20T03:36:13.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2600000000000002e-07 2023-10-20T03:36:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898832+0000", 2023-10-20T03:36:13.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058099900000000001 2023-10-20T03:36:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898866+0000", 2023-10-20T03:36:13.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4060000000000003e-05 2023-10-20T03:36:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898894+0000", 2023-10-20T03:36:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7538999999999998e-05 2023-10-20T03:36:13.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899124+0000", 2023-10-20T03:36:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023028600000000001 2023-10-20T03:36:13.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899154+0000", 2023-10-20T03:36:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9485999999999999e-05 2023-10-20T03:36:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926698+0000", 2023-10-20T03:36:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027544361 2023-10-20T03:36:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.926777+0000", 2023-10-20T03:36:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8769000000000004e-05 2023-10-20T03:36:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.014 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:556 3.3s0 3:c0108eeb:::smithi17039300-21:head [append 0~638976] snapc 23=[23] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:36:13.015 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.956395+0000", 2023-10-20T03:36:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.018189981999999, 2023-10-20T03:36:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039551132000000003, 2023-10-20T03:36:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 556 2023-10-20T03:36:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.956395+0000", 2023-10-20T03:36:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.956395+0000", 2023-10-20T03:36:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.956398+0000", 2023-10-20T03:36:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.903e-06 2023-10-20T03:36:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.957575+0000", 2023-10-20T03:36:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001176912 2023-10-20T03:36:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.957610+0000", 2023-10-20T03:36:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4656000000000002e-05 2023-10-20T03:36:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.957626+0000", 2023-10-20T03:36:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6676000000000001e-05 2023-10-20T03:36:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.957768+0000", 2023-10-20T03:36:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141975 2023-10-20T03:36:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2023-10-20T03:36:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.957785+0000", 2023-10-20T03:36:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6612999999999998e-05 2023-10-20T03:36:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.963559+0000", 2023-10-20T03:36:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057735659999999999 2023-10-20T03:36:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.963781+0000", 2023-10-20T03:36:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022263 2023-10-20T03:36:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966469+0000", 2023-10-20T03:36:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026878179999999998 2023-10-20T03:36:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.983765+0000", 2023-10-20T03:36:13.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017296308 2023-10-20T03:36:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.995677+0000", 2023-10-20T03:36:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011912109000000001 2023-10-20T03:36:13.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.995946+0000", 2023-10-20T03:36:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026896599999999999 2023-10-20T03:36:13.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.031 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:557 3.3s0 3:c0108eeb:::smithi17039300-21:head [append 0~458752] snapc 23=[23] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:36:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.963864+0000", 2023-10-20T03:36:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.010721566000001, 2023-10-20T03:36:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050243923000000003, 2023-10-20T03:36:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.032 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 557 2023-10-20T03:36:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.033 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.963864+0000", 2023-10-20T03:36:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.963864+0000", 2023-10-20T03:36:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.963866+0000", 2023-10-20T03:36:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0169999999999999e-06 2023-10-20T03:36:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.964523+0000", 2023-10-20T03:36:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065709600000000003 2023-10-20T03:36:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.964548+0000", 2023-10-20T03:36:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5032999999999999e-05 2023-10-20T03:36:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.964588+0000", 2023-10-20T03:36:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9991999999999999e-05 2023-10-20T03:36:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.984454+0000", 2023-10-20T03:36:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019866051999999999 2023-10-20T03:36:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.984811+0000", 2023-10-20T03:36:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035713500000000002 2023-10-20T03:36:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.986904+0000", 2023-10-20T03:36:13.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002092867 2023-10-20T03:36:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.013182+0000", 2023-10-20T03:36:13.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026277805000000001 2023-10-20T03:36:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.013831+0000", 2023-10-20T03:36:13.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064906199999999995 2023-10-20T03:36:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.014108+0000", 2023-10-20T03:36:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027686399999999998 2023-10-20T03:36:13.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:558 3.3s0 3:c0108eeb:::smithi17039300-21:head [append 0~704512] snapc 23=[23] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:36:13.045 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.975672+0000", 2023-10-20T03:36:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.998913405, 2023-10-20T03:36:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039948122000000003, 2023-10-20T03:36:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 558 2023-10-20T03:36:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.047 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.975672+0000", 2023-10-20T03:36:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.975672+0000", 2023-10-20T03:36:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.975675+0000", 2023-10-20T03:36:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4999999999999999e-06 2023-10-20T03:36:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977096+0000", 2023-10-20T03:36:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014207480000000001 2023-10-20T03:36:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977137+0000", 2023-10-20T03:36:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1149000000000002e-05 2023-10-20T03:36:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977159+0000", 2023-10-20T03:36:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2269999999999999e-05 2023-10-20T03:36:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.996676+0000", 2023-10-20T03:36:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019517016000000002 2023-10-20T03:36:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.996904+0000", 2023-10-20T03:36:13.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022794699999999999 2023-10-20T03:36:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.998600+0000", 2023-10-20T03:36:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001695237 2023-10-20T03:36:13.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015338+0000", 2023-10-20T03:36:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016738376999999999 2023-10-20T03:36:13.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015447+0000", 2023-10-20T03:36:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010899 2023-10-20T03:36:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015620+0000", 2023-10-20T03:36:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172888 2023-10-20T03:36:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:559 3.3s0 3:c0108eeb:::smithi17039300-21:head [setxattr _header (57) in=7b] snapc 23=[23] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:36:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.977198+0000", 2023-10-20T03:36:13.059 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.997387229000001, 2023-10-20T03:36:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039461151999999999, 2023-10-20T03:36:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 559 2023-10-20T03:36:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977198+0000", 2023-10-20T03:36:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977198+0000", 2023-10-20T03:36:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977200+0000", 2023-10-20T03:36:13.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7600000000000001e-06 2023-10-20T03:36:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977225+0000", 2023-10-20T03:36:13.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4817e-05 2023-10-20T03:36:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977251+0000", 2023-10-20T03:36:13.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6139000000000002e-05 2023-10-20T03:36:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977268+0000", 2023-10-20T03:36:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7076e-05 2023-10-20T03:36:13.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.011294+0000", 2023-10-20T03:36:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034025979999999997 2023-10-20T03:36:13.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.011452+0000", 2023-10-20T03:36:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000158639 2023-10-20T03:36:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.012364+0000", 2023-10-20T03:36:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00091161599999999997 2023-10-20T03:36:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015628+0000", 2023-10-20T03:36:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032641079999999999 2023-10-20T03:36:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015729+0000", 2023-10-20T03:36:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101335 2023-10-20T03:36:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.016659+0000", 2023-10-20T03:36:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00092968200000000003 2023-10-20T03:36:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:560 3.3s0 3:c0108eeb:::smithi17039300-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:36:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.977299+0000", 2023-10-20T03:36:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.997286076999998, 2023-10-20T03:36:13.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043501639000000002, 2023-10-20T03:36:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:36:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.074 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 560 2023-10-20T03:36:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977299+0000", 2023-10-20T03:36:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977299+0000", 2023-10-20T03:36:13.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977300+0000", 2023-10-20T03:36:13.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.373e-06 2023-10-20T03:36:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977307+0000", 2023-10-20T03:36:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.387e-06 2023-10-20T03:36:13.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977330+0000", 2023-10-20T03:36:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3359000000000001e-05 2023-10-20T03:36:13.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.977346+0000", 2023-10-20T03:36:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6093999999999999e-05 2023-10-20T03:36:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015155+0000", 2023-10-20T03:36:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037808698000000002 2023-10-20T03:36:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.015327+0000", 2023-10-20T03:36:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171554 2023-10-20T03:36:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.016792+0000", 2023-10-20T03:36:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014655930000000001 2023-10-20T03:36:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.017045+0000", 2023-10-20T03:36:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025238700000000002 2023-10-20T03:36:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.020801+0000", 2023-10-20T03:36:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003756194 2023-10-20T03:36:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:575 3.3s0 3:c87f890b:::smithi17039300-50:head [append 0~704512] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.979848+0000", 2023-10-20T03:36:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.994737167, 2023-10-20T03:36:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040944178999999997, 2023-10-20T03:36:13.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.087 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 575 2023-10-20T03:36:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.979848+0000", 2023-10-20T03:36:13.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.979848+0000", 2023-10-20T03:36:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.979850+0000", 2023-10-20T03:36:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2539999999999999e-06 2023-10-20T03:36:13.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.981043+0000", 2023-10-20T03:36:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011924699999999999 2023-10-20T03:36:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.981087+0000", 2023-10-20T03:36:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3968999999999999e-05 2023-10-20T03:36:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.981101+0000", 2023-10-20T03:36:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4722999999999999e-05 2023-10-20T03:36:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.981240+0000", 2023-10-20T03:36:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013845799999999999 2023-10-20T03:36:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.981460+0000", 2023-10-20T03:36:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022006599999999999 2023-10-20T03:36:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.983567+0000", 2023-10-20T03:36:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021068749999999998 2023-10-20T03:36:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.019744+0000", 2023-10-20T03:36:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036176940999999997 2023-10-20T03:36:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.020535+0000", 2023-10-20T03:36:13.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079162700000000004 2023-10-20T03:36:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.020792+0000", 2023-10-20T03:36:13.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000256796 2023-10-20T03:36:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:576 3.3s0 3:c87f890b:::smithi17039300-50:head [append 0~655360] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.984983+0000", 2023-10-20T03:36:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.989602265999999, 2023-10-20T03:36:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.057738351, 2023-10-20T03:36:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.100 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 576 2023-10-20T03:36:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.101 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984983+0000", 2023-10-20T03:36:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984983+0000", 2023-10-20T03:36:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.984986+0000", 2023-10-20T03:36:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7499999999999999e-06 2023-10-20T03:36:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.985845+0000", 2023-10-20T03:36:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085890599999999999 2023-10-20T03:36:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.985879+0000", 2023-10-20T03:36:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4016000000000001e-05 2023-10-20T03:36:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.985901+0000", 2023-10-20T03:36:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2163000000000001e-05 2023-10-20T03:36:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.002888+0000", 2023-10-20T03:36:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016987001000000002 2023-10-20T03:36:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.003224+0000", 2023-10-20T03:36:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000335908 2023-10-20T03:36:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.006382+0000", 2023-10-20T03:36:13.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031583039999999998 2023-10-20T03:36:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.021138+0000", 2023-10-20T03:36:13.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014756438 2023-10-20T03:36:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.042493+0000", 2023-10-20T03:36:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021354610999999999 2023-10-20T03:36:13.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.042721+0000", 2023-10-20T03:36:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000228254 2023-10-20T03:36:13.112 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:577 3.3s0 3:c87f890b:::smithi17039300-50:head [append 0~507904] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:13.113 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.988803+0000", 2023-10-20T03:36:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.985782480000001, 2023-10-20T03:36:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054481475000000001, 2023-10-20T03:36:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.114 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 577 2023-10-20T03:36:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.988803+0000", 2023-10-20T03:36:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.988803+0000", 2023-10-20T03:36:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.988805+0000", 2023-10-20T03:36:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.793e-06 2023-10-20T03:36:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989567+0000", 2023-10-20T03:36:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076297099999999998 2023-10-20T03:36:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989595+0000", 2023-10-20T03:36:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7195999999999998e-05 2023-10-20T03:36:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989614+0000", 2023-10-20T03:36:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.931e-05 2023-10-20T03:36:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.021402+0000", 2023-10-20T03:36:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031787718999999999 2023-10-20T03:36:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.021783+0000", 2023-10-20T03:36:13.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000381648 2023-10-20T03:36:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.024185+0000", 2023-10-20T03:36:13.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002401612 2023-10-20T03:36:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.041211+0000", 2023-10-20T03:36:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017025795 2023-10-20T03:36:13.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.043097+0000", 2023-10-20T03:36:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018863269999999999 2023-10-20T03:36:13.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.043284+0000", 2023-10-20T03:36:13.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000187104 2023-10-20T03:36:13.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.126 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.127 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.127 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:578 3.3s0 3:c87f890b:::smithi17039300-50:head [setxattr _header (57) in=7b] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:13.127 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.989649+0000", 2023-10-20T03:36:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.984935926999999, 2023-10-20T03:36:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054676374999999999, 2023-10-20T03:36:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.128 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 578 2023-10-20T03:36:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989649+0000", 2023-10-20T03:36:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989649+0000", 2023-10-20T03:36:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989651+0000", 2023-10-20T03:36:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4559999999999999e-06 2023-10-20T03:36:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989662+0000", 2023-10-20T03:36:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1077000000000001e-05 2023-10-20T03:36:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989684+0000", 2023-10-20T03:36:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2189e-05 2023-10-20T03:36:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989701+0000", 2023-10-20T03:36:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6750000000000001e-05 2023-10-20T03:36:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.038393+0000", 2023-10-20T03:36:13.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048692566 2023-10-20T03:36:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.038741+0000", 2023-10-20T03:36:13.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000347315 2023-10-20T03:36:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.040161+0000", 2023-10-20T03:36:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014204409999999999 2023-10-20T03:36:13.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.041971+0000", 2023-10-20T03:36:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001809532 2023-10-20T03:36:13.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.044136+0000", 2023-10-20T03:36:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021652749999999999 2023-10-20T03:36:13.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.044326+0000", 2023-10-20T03:36:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000189774 2023-10-20T03:36:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.140 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:579 3.3s0 3:c87f890b:::smithi17039300-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.989733+0000", 2023-10-20T03:36:13.141 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.984852232000001, 2023-10-20T03:36:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058151636, 2023-10-20T03:36:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:36:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 579 2023-10-20T03:36:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989733+0000", 2023-10-20T03:36:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989733+0000", 2023-10-20T03:36:13.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989734+0000", 2023-10-20T03:36:13.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.406e-06 2023-10-20T03:36:13.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989742+0000", 2023-10-20T03:36:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3270000000000003e-06 2023-10-20T03:36:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989761+0000", 2023-10-20T03:36:13.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9326000000000001e-05 2023-10-20T03:36:13.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.989778+0000", 2023-10-20T03:36:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.649e-05 2023-10-20T03:36:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.045369+0000", 2023-10-20T03:36:13.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055591030999999999 2023-10-20T03:36:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.045545+0000", 2023-10-20T03:36:13.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176732 2023-10-20T03:36:13.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.047885+0000", 2023-10-20T03:36:13.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023393239999999998 2023-10-20T03:36:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.152 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 71/56 ECSubWriteReply(tid=63, last_complete=71'46, committed=1, applied=1))", 2023-10-20T03:36:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:52.995405+0000", 2023-10-20T03:36:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.979180542999998, 2023-10-20T03:36:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024140920999999999, 2023-10-20T03:36:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995405+0000", 2023-10-20T03:36:13.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995405+0000", 2023-10-20T03:36:13.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995406+0000", 2023-10-20T03:36:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3599999999999999e-06 2023-10-20T03:36:13.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995416+0000", 2023-10-20T03:36:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9029999999999999e-06 2023-10-20T03:36:13.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995446+0000", 2023-10-20T03:36:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0358999999999999e-05 2023-10-20T03:36:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:52.995458+0000", 2023-10-20T03:36:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1507e-05 2023-10-20T03:36:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.019489+0000", 2023-10-20T03:36:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024030969999999999 2023-10-20T03:36:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.019546+0000", 2023-10-20T03:36:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6821999999999999e-05 2023-10-20T03:36:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:626 3.3s0 3:c87f890b:::smithi17039300-50:head [read 0~4259840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2023-10-20T03:36:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:57.010115+0000", 2023-10-20T03:36:13.162 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.964469831000001, 2023-10-20T03:36:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024359861999999999, 2023-10-20T03:36:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:36:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.163 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 626 2023-10-20T03:36:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010115+0000", 2023-10-20T03:36:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010115+0000", 2023-10-20T03:36:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010116+0000", 2023-10-20T03:36:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6300000000000004e-07 2023-10-20T03:36:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010121+0000", 2023-10-20T03:36:13.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1320000000000002e-06 2023-10-20T03:36:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010136+0000", 2023-10-20T03:36:13.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5133000000000001e-05 2023-10-20T03:36:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010144+0000", 2023-10-20T03:36:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0199999999999997e-06 2023-10-20T03:36:13.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010325+0000", 2023-10-20T03:36:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018146899999999999 2023-10-20T03:36:13.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010338+0000", 2023-10-20T03:36:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3046e-05 2023-10-20T03:36:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.014721+0000", 2023-10-20T03:36:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043831449999999997 2023-10-20T03:36:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.014841+0000", 2023-10-20T03:36:13.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119465 2023-10-20T03:36:13.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.034475+0000", 2023-10-20T03:36:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019634589000000001 2023-10-20T03:36:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:627 3.3s0 3:c87f890b:::smithi17039300-50:head [read 0~4259840,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2023-10-20T03:36:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:57.010173+0000", 2023-10-20T03:36:13.175 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.964411706, 2023-10-20T03:36:13.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027277798999999998, 2023-10-20T03:36:13.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:36:13.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 627 2023-10-20T03:36:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010173+0000", 2023-10-20T03:36:13.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010173+0000", 2023-10-20T03:36:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010174+0000", 2023-10-20T03:36:13.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6899999999999997e-07 2023-10-20T03:36:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010179+0000", 2023-10-20T03:36:13.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1120000000000004e-06 2023-10-20T03:36:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010191+0000", 2023-10-20T03:36:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1817999999999999e-05 2023-10-20T03:36:13.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010200+0000", 2023-10-20T03:36:13.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8820000000000007e-06 2023-10-20T03:36:13.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010388+0000", 2023-10-20T03:36:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188207 2023-10-20T03:36:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.010400+0000", 2023-10-20T03:36:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1486e-05 2023-10-20T03:36:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.019217+0000", 2023-10-20T03:36:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088170820000000004 2023-10-20T03:36:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.019310+0000", 2023-10-20T03:36:13.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3078999999999994e-05 2023-10-20T03:36:13.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:57.037451+0000", 2023-10-20T03:36:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018141464 2023-10-20T03:36:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s3 77/31 ECSubWrite(tid=303, reqid=client.4570.0:632, at_version=77'73, trim_to=0'0, roll_forward_to=68'72))", 2023-10-20T03:36:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:58.786377+0000", 2023-10-20T03:36:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.188208514999999, 2023-10-20T03:36:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024614448000000001, 2023-10-20T03:36:13.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.786377+0000", 2023-10-20T03:36:13.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.786377+0000", 2023-10-20T03:36:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.786377+0000", 2023-10-20T03:36:13.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8000000000000001e-07 2023-10-20T03:36:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.787263+0000", 2023-10-20T03:36:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088630700000000002 2023-10-20T03:36:13.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.787293+0000", 2023-10-20T03:36:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9505999999999998e-05 2023-10-20T03:36:13.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.787309+0000", 2023-10-20T03:36:13.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.592e-05 2023-10-20T03:36:13.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.787469+0000", 2023-10-20T03:36:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016006700000000001 2023-10-20T03:36:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.787487+0000", 2023-10-20T03:36:13.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8383999999999999e-05 2023-10-20T03:36:13.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.810891+0000", 2023-10-20T03:36:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023403688999999998 2023-10-20T03:36:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.810991+0000", 2023-10-20T03:36:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100195 2023-10-20T03:36:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:704 3.3s0 3:c87f890b:::smithi17039300-50:head [read 0~6324224,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2023-10-20T03:36:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:07.062769+0000", 2023-10-20T03:36:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.9118163089999998, 2023-10-20T03:36:13.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029636541999999998, 2023-10-20T03:36:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:36:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.200 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 704 2023-10-20T03:36:13.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.201 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062769+0000", 2023-10-20T03:36:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062769+0000", 2023-10-20T03:36:13.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062772+0000", 2023-10-20T03:36:13.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8760000000000001e-06 2023-10-20T03:36:13.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062787+0000", 2023-10-20T03:36:13.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5073e-05 2023-10-20T03:36:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062820+0000", 2023-10-20T03:36:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3506000000000001e-05 2023-10-20T03:36:13.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062838+0000", 2023-10-20T03:36:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7736000000000001e-05 2023-10-20T03:36:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.063037+0000", 2023-10-20T03:36:13.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000198774 2023-10-20T03:36:13.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2023-10-20T03:36:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.063056+0000", 2023-10-20T03:36:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9193000000000001e-05 2023-10-20T03:36:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.070064+0000", 2023-10-20T03:36:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070077749999999999 2023-10-20T03:36:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.070370+0000", 2023-10-20T03:36:13.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030644400000000001 2023-10-20T03:36:13.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.092405+0000", 2023-10-20T03:36:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022035164999999999 2023-10-20T03:36:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.212 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:705 3.3s0 3:c87f890b:::smithi17039300-50:head [read 0~6324224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2023-10-20T03:36:13.212 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:07.062881+0000", 2023-10-20T03:36:13.212 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.9117046559999995, 2023-10-20T03:36:13.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037736532000000003, 2023-10-20T03:36:13.212 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:36:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.213 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 705 2023-10-20T03:36:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062881+0000", 2023-10-20T03:36:13.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062881+0000", 2023-10-20T03:36:13.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062881+0000", 2023-10-20T03:36:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0400000000000005e-07 2023-10-20T03:36:13.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062887+0000", 2023-10-20T03:36:13.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4500000000000003e-06 2023-10-20T03:36:13.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062899+0000", 2023-10-20T03:36:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2072000000000001e-05 2023-10-20T03:36:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062907+0000", 2023-10-20T03:36:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4430000000000003e-06 2023-10-20T03:36:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.063204+0000", 2023-10-20T03:36:13.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029669499999999998 2023-10-20T03:36:13.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.063217+0000", 2023-10-20T03:36:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2526e-05 2023-10-20T03:36:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.071418+0000", 2023-10-20T03:36:13.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082012130000000006 2023-10-20T03:36:13.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.071718+0000", 2023-10-20T03:36:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000300484 2023-10-20T03:36:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.100617+0000", 2023-10-20T03:36:13.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028898844999999999 2023-10-20T03:36:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:706 3.3s0 3:c87f890b:::smithi17039300-50:head [read 0~6324224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2023-10-20T03:36:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:07.062994+0000", 2023-10-20T03:36:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.9115908990000001, 2023-10-20T03:36:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043823964, 2023-10-20T03:36:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.225 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:36:13.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.226 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 706 2023-10-20T03:36:13.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062994+0000", 2023-10-20T03:36:13.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062994+0000", 2023-10-20T03:36:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.062995+0000", 2023-10-20T03:36:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8999999999999995e-07 2023-10-20T03:36:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.063004+0000", 2023-10-20T03:36:13.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7860000000000002e-06 2023-10-20T03:36:13.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.063017+0000", 2023-10-20T03:36:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.294e-05 2023-10-20T03:36:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.063029+0000", 2023-10-20T03:36:13.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1793e-05 2023-10-20T03:36:13.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.063281+0000", 2023-10-20T03:36:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025225900000000001 2023-10-20T03:36:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:13.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.063295+0000", 2023-10-20T03:36:13.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.429e-05 2023-10-20T03:36:13.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.072684+0000", 2023-10-20T03:36:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0093888070000000007 2023-10-20T03:36:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.073049+0000", 2023-10-20T03:36:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036520200000000002 2023-10-20T03:36:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:07.106818+0000", 2023-10-20T03:36:13.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033768997000000002 2023-10-20T03:36:13.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.237 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:36:13.239 INFO:tasks.rados.rados.0.smithi170.stdout:317: done (1 left) 2023-10-20T03:36:13.239 INFO:tasks.rados.rados.0.smithi170.stdout:318: rollback oid 23 current snap is 30 2023-10-20T03:36:13.239 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 23 to 27 2023-10-20T03:36:13.240 INFO:tasks.rados.rados.0.smithi170.stdout:319: read oid 25 snap 27 2023-10-20T03:36:13.240 INFO:tasks.rados.rados.0.smithi170.stdout:319: expect (ObjNum 100 snap 13 seq_num 100) 2023-10-20T03:36:13.240 INFO:tasks.rados.rados.0.smithi170.stdout:320: rollback oid 39 current snap is 30 2023-10-20T03:36:13.240 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 39 to 27 2023-10-20T03:36:13.240 INFO:tasks.rados.rados.0.smithi170.stdout:321: copy_from oid 48 from oid 1 current snap is 30 2023-10-20T03:36:13.241 INFO:tasks.rados.rados.0.smithi170.stdout:322: snap_remove snap 26 2023-10-20T03:36:13.241 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:13.233+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:13.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-10-20T03:36:13.340 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:13.336+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:13.387 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:36:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:36:13.387 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2023-10-20T03:36:13.387 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:36:13.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-10-20T03:36:13.448 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:13.447+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:13.540 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:13.537+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:13.546 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:36:13.547 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2023-10-20T03:36:13.547 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2023-10-20T03:36:13.547 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2023-10-20T03:36:13.547 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:36:13.562 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-10-20T03:36:13.648 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:13.647+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:13.743 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:36:13.743 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2023-10-20T03:36:13.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2023-10-20T03:36:13.744 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2023-10-20T03:36:13.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.744 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:36:13.744 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.329666+0000", 2023-10-20T03:36:13.744 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.410054725000002, 2023-10-20T03:36:13.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026825213000000001, 2023-10-20T03:36:13.745 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.745 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.745 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329666+0000", 2023-10-20T03:36:13.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329666+0000", 2023-10-20T03:36:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329667+0000", 2023-10-20T03:36:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2023-10-20T03:36:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330048+0000", 2023-10-20T03:36:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038081299999999997 2023-10-20T03:36:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330082+0000", 2023-10-20T03:36:13.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4402e-05 2023-10-20T03:36:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330100+0000", 2023-10-20T03:36:13.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7683e-05 2023-10-20T03:36:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330268+0000", 2023-10-20T03:36:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016784499999999999 2023-10-20T03:36:13.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330296+0000", 2023-10-20T03:36:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8367000000000001e-05 2023-10-20T03:36:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356426+0000", 2023-10-20T03:36:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026129432000000001 2023-10-20T03:36:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356492+0000", 2023-10-20T03:36:13.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6063999999999996e-05 2023-10-20T03:36:13.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.754 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s1 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:36:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.357167+0000", 2023-10-20T03:36:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.38255384, 2023-10-20T03:36:13.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025761703, 2023-10-20T03:36:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357167+0000", 2023-10-20T03:36:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357167+0000", 2023-10-20T03:36:13.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357169+0000", 2023-10-20T03:36:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.967e-06 2023-10-20T03:36:13.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357532+0000", 2023-10-20T03:36:13.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036283700000000002 2023-10-20T03:36:13.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357564+0000", 2023-10-20T03:36:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1798999999999998e-05 2023-10-20T03:36:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357615+0000", 2023-10-20T03:36:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1004999999999997e-05 2023-10-20T03:36:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357832+0000", 2023-10-20T03:36:13.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000216608 2023-10-20T03:36:13.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357857+0000", 2023-10-20T03:36:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5092999999999998e-05 2023-10-20T03:36:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.382875+0000", 2023-10-20T03:36:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025017978999999999 2023-10-20T03:36:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.382929+0000", 2023-10-20T03:36:13.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4415e-05 2023-10-20T03:36:13.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.765 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:320 3.2s0 3:474cd252:::smithi17039300-42:head [create,setxattr c[X]j^UsOa`q]sYrSaa]raqNm]iSd^X]OmelObeN (242) in=40b,setxattr gopkVbealbs^Z^^`^sXjNlPPYp^\\fbk\\fUnmTq[R (208) in=40b,setxattr eUZoNnss (4560) in=8b,setxattr SYo`knsRbgPQPPbXW[[ZsTehoXbSPY_WQl]_URc` (105) in=40b,setxattr \\PYbW]PkbTb^sYhsihYgqj]pROo]\\NWqNhT_lWgn (17753) in=40b,setxattr ]R_fObdq\\NUZ]QiO]j`djkVVp`bfRdph]Zbea]gg (17332) in=40b,setxattr ]ptcajatsVQWlWeoNNlPngWealknZSiQarONqbaV (1473) in=40b,setxattr `i_qfTQWSto[gYWTmYsOtOhsW[fnhaVacqaadrU] (1191) in=40b,setxattr nNhpa\\e^VfUjte_Pc[kSor^[gpr[eaflXRs]SWZr (16140) in=40b,setxattr aRNZ_mZpadm^l[rZntn`UX^RN\\cZ_TT]rRTjibjc (1121) in=40b,setxattr aVipjRkr_^aolT[aa`UQhsfb\\eRQ[ZZleb_OQOlX (2384) in=40b,setxattr m^VcgrkhPRmZlXiSggOaoPTi^]^WnQh\\aXaekmoc (947) in=40b,setxattr NikdcN_ilNkWgOQgZqrYPT\\b[drPRq\\niPPQdbVU (5260) in=40b,setxattr ajnWlVTfP^PVRs[VchU^kT_gSmcjmZRh]ofVgTrp (1437) in=40b,setxattr liRPRT`aa[cqbZkpaWa\\q[NZWUXYk[a[cicigTsV (1017) in=40b,setxattr XkR_Nkkpdeiacga[]Y^NismNTd`tQP[eb[qbrrd^ (8912) in=40b,setxattr ]mai`f_WjqaYertRarRZ[mrmWf_`mktRW`slld^_ (243) in=40b,setxattr jacQPeR\\bNj_nOTOc^WkPrSpceTds[[eee`SndWY (1086) in=40b,setxattr \\lcQhm[Vm\\jW\\cPdSlWoZcU[d^htQTrhRak`t\\hk (4068) in=40b,setxattr e`Od\\\\V]jhaQVknTVdWZOTq[eYbmlehRf^ZtoZnQ (384) in=40b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:26.333521+0000", 2023-10-20T03:36:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.406199706999999, 2023-10-20T03:36:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016194731, 2023-10-20T03:36:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.767 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 320 2023-10-20T03:36:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333521+0000", 2023-10-20T03:36:13.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333521+0000", 2023-10-20T03:36:13.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333524+0000", 2023-10-20T03:36:13.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8270000000000002e-06 2023-10-20T03:36:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333727+0000", 2023-10-20T03:36:13.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000202504 2023-10-20T03:36:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333762+0000", 2023-10-20T03:36:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5271999999999997e-05 2023-10-20T03:36:13.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333780+0000", 2023-10-20T03:36:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8280000000000001e-05 2023-10-20T03:36:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333968+0000", 2023-10-20T03:36:13.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000187655 2023-10-20T03:36:13.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2023-10-20T03:36:13.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.333984+0000", 2023-10-20T03:36:13.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5739e-05 2023-10-20T03:36:13.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.339320+0000", 2023-10-20T03:36:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0053365089999999997 2023-10-20T03:36:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.339685+0000", 2023-10-20T03:36:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036516599999999999 2023-10-20T03:36:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.342206+0000", 2023-10-20T03:36:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0025201440000000002 2023-10-20T03:36:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.346469+0000", 2023-10-20T03:36:13.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042632420000000004 2023-10-20T03:36:13.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.349492+0000", 2023-10-20T03:36:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030232560000000002 2023-10-20T03:36:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:26.349716+0000", 2023-10-20T03:36:13.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022413700000000001 2023-10-20T03:36:13.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.781 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.781 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:387 3.2s0 3:4f9c85d4:::smithi17039300-39:d [read 0~1622016,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:36:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.381906+0000", 2023-10-20T03:36:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.357815166999998, 2023-10-20T03:36:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016689273000000001, 2023-10-20T03:36:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.782 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:36:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 387 2023-10-20T03:36:13.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381906+0000", 2023-10-20T03:36:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381906+0000", 2023-10-20T03:36:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381909+0000", 2023-10-20T03:36:13.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.965e-06 2023-10-20T03:36:13.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381925+0000", 2023-10-20T03:36:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5914999999999999e-05 2023-10-20T03:36:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381953+0000", 2023-10-20T03:36:13.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8430999999999999e-05 2023-10-20T03:36:13.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.381968+0000", 2023-10-20T03:36:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4430000000000001e-05 2023-10-20T03:36:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382204+0000", 2023-10-20T03:36:13.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023644700000000001 2023-10-20T03:36:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2023-10-20T03:36:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382224+0000", 2023-10-20T03:36:13.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9406e-05 2023-10-20T03:36:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.387100+0000", 2023-10-20T03:36:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048765429999999997 2023-10-20T03:36:13.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.387279+0000", 2023-10-20T03:36:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017912599999999999 2023-10-20T03:36:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.398595+0000", 2023-10-20T03:36:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01131601 2023-10-20T03:36:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.793 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.794 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.794 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:388 3.2s0 3:4f9c85d4:::smithi17039300-39:d [read 0~1622016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:36:13.794 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.382005+0000", 2023-10-20T03:36:13.795 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.357716547999999, 2023-10-20T03:36:13.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019238755, 2023-10-20T03:36:13.795 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.795 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:36:13.795 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 388 2023-10-20T03:36:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382005+0000", 2023-10-20T03:36:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382005+0000", 2023-10-20T03:36:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382005+0000", 2023-10-20T03:36:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4200000000000005e-07 2023-10-20T03:36:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382011+0000", 2023-10-20T03:36:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0839999999999999e-06 2023-10-20T03:36:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382020+0000", 2023-10-20T03:36:13.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8709999999999992e-06 2023-10-20T03:36:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382029+0000", 2023-10-20T03:36:13.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0450000000000006e-06 2023-10-20T03:36:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382364+0000", 2023-10-20T03:36:13.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000334493 2023-10-20T03:36:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382374+0000", 2023-10-20T03:36:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.028e-05 2023-10-20T03:36:13.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.388188+0000", 2023-10-20T03:36:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058138490000000003 2023-10-20T03:36:13.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.388305+0000", 2023-10-20T03:36:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116747 2023-10-20T03:36:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.401243+0000", 2023-10-20T03:36:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012938544 2023-10-20T03:36:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.806 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:389 3.2s0 3:4f9c85d4:::smithi17039300-39:d [read 0~1622016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:36:13.807 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:31.382042+0000", 2023-10-20T03:36:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.357679588000003, 2023-10-20T03:36:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021763228999999999, 2023-10-20T03:36:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:36:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.808 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 389 2023-10-20T03:36:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382042+0000", 2023-10-20T03:36:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382042+0000", 2023-10-20T03:36:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382042+0000", 2023-10-20T03:36:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8899999999999999e-07 2023-10-20T03:36:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382046+0000", 2023-10-20T03:36:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0319999999999997e-06 2023-10-20T03:36:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382061+0000", 2023-10-20T03:36:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4800999999999999e-05 2023-10-20T03:36:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382067+0000", 2023-10-20T03:36:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0630000000000002e-06 2023-10-20T03:36:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382430+0000", 2023-10-20T03:36:13.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000362505 2023-10-20T03:36:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.382439+0000", 2023-10-20T03:36:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2199999999999998e-06 2023-10-20T03:36:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.388649+0000", 2023-10-20T03:36:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0062098459999999998 2023-10-20T03:36:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.388756+0000", 2023-10-20T03:36:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107316 2023-10-20T03:36:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:31.403805+0000", 2023-10-20T03:36:13.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015048857000000001 2023-10-20T03:36:13.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.820 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.820 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs3 53/47 ECSubWrite(tid=58, reqid=client.4570.0:409, at_version=53'32, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:13.820 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:35.472767+0000", 2023-10-20T03:36:13.821 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.266954304999999, 2023-10-20T03:36:13.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022685556999999999, 2023-10-20T03:36:13.821 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.821 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.821 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472767+0000", 2023-10-20T03:36:13.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472767+0000", 2023-10-20T03:36:13.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.472769+0000", 2023-10-20T03:36:13.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6869999999999999e-06 2023-10-20T03:36:13.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473140+0000", 2023-10-20T03:36:13.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037175199999999999 2023-10-20T03:36:13.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473171+0000", 2023-10-20T03:36:13.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0855000000000002e-05 2023-10-20T03:36:13.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473186+0000", 2023-10-20T03:36:13.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4946999999999999e-05 2023-10-20T03:36:13.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473343+0000", 2023-10-20T03:36:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015689 2023-10-20T03:36:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.473369+0000", 2023-10-20T03:36:13.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6389000000000001e-05 2023-10-20T03:36:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.495356+0000", 2023-10-20T03:36:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021986362999999998 2023-10-20T03:36:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:35.495452+0000", 2023-10-20T03:36:13.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6674000000000003e-05 2023-10-20T03:36:13.830 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:433 3.2s0 3:474cd252:::smithi17039300-42:head [create,setxattr [VYQXaeer[Nsq`eTbgjNZfdhqbTRn_][lloSeZUP (16840) in=40b,setxattr hpaXc^]qjrqU_ciY[cWncm]SPa^tP_jRrRXm`jol (175) in=40b,setxattr YVpShpT[fhfamkUsgr]mb\\foasmX^k`jdhbbdlfP (1823) in=40b,setxattr ZhfkQl`OllQo^hemUnadbmqWdoabfgjcc^sVm_gf (1611) in=40b,setxattr aakOOaWemYRPYcaZ\\knQb (1051) in=21b,setxattr gPaagTUadWkOtqaRt]^rdUj^Zinmt]U^ (1051) in=32b,setxattr SfXmSZ]crX[dqcaqjNrrroQQbptW^Q^nUSUlcgdk (807) in=40b,setxattr SmRaehVPpUNpfZ[UX\\ZO_Ygfofk^ah^Xki^XTeX` (1191) in=40b,setxattr aTgW^YtXrjnrhT (1051) in=14b,setxattr idO[Vrlsc^pW[hYfh`UQSs^m_mU[YefWeld[YZ[c (1543) in=40b,setxattr ]Q``task\\rr]qaY[Pg`i[alh[rPlSdrcQ^iW^Z^g (1157) in=40b,setxattr _OOWpoQQg\\jtQUi``q_s[ae_akq`YOYilNTq^SPO (1121) in=40b,setxattr kN\\_ZaOX]q[kXen^Z]liYfaVP]prk\\r\\\\_Wsd[d_ (105) in=40b,setxattr OU`jdr^jTir[joOegmm (1051) in=19b,setxattr tRofbjROgtWqoRfmUVZXhpaZ[Q^sUNZb`\\YsdUfh (1507) in=40b,setxattr RRNpbt[`okmbR]RpSWfmmedXrUpaltcbWPbNRZRZ (67) in=40b,setxattr i[`^r^aojXRaU[[T\\kgWN^ (1051) in=22b,setxattr gZhSajUgdm`o]W[X`sgdVl^TtY^_SX[_l[Tr_aaO (663) in=40b,setxattr qN][QajljeddtQTcQX_QaTrg\\V\\pY\\QbsWehde`s (1646) in=40b,setxattr fbTrNt\\`ZVPVN[]tcg]iPXYOnmYnV]oaW_^o^gdr (1403) in=40b] snapc 16=[16,3,2] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2023-10-20T03:36:13.831 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:37.441170+0000", 2023-10-20T03:36:13.832 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.298551373999999, 2023-10-20T03:36:13.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020595808, 2023-10-20T03:36:13.832 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.832 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.832 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 433 2023-10-20T03:36:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441170+0000", 2023-10-20T03:36:13.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441170+0000", 2023-10-20T03:36:13.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441172+0000", 2023-10-20T03:36:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.469e-06 2023-10-20T03:36:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441400+0000", 2023-10-20T03:36:13.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000227226 2023-10-20T03:36:13.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441441+0000", 2023-10-20T03:36:13.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1644999999999999e-05 2023-10-20T03:36:13.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441467+0000", 2023-10-20T03:36:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5412999999999999e-05 2023-10-20T03:36:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441665+0000", 2023-10-20T03:36:13.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019792900000000001 2023-10-20T03:36:13.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.441999+0000", 2023-10-20T03:36:13.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000334073 2023-10-20T03:36:13.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.445086+0000", 2023-10-20T03:36:13.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030878160000000002 2023-10-20T03:36:13.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.460222+0000", 2023-10-20T03:36:13.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015135984 2023-10-20T03:36:13.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.460491+0000", 2023-10-20T03:36:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026856699999999999 2023-10-20T03:36:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.461766+0000", 2023-10-20T03:36:13.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012746859999999999 2023-10-20T03:36:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.845 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.fs3 55/47 ECSubWrite(tid=71, reqid=client.4570.0:406, at_version=55'38, trim_to=0'0, roll_forward_to=54'36))", 2023-10-20T03:36:13.846 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:37.472200+0000", 2023-10-20T03:36:13.846 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.267521416999998, 2023-10-20T03:36:13.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02292278, 2023-10-20T03:36:13.846 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.846 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.847 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.472200+0000", 2023-10-20T03:36:13.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.472200+0000", 2023-10-20T03:36:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.472201+0000", 2023-10-20T03:36:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3230000000000001e-06 2023-10-20T03:36:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.473418+0000", 2023-10-20T03:36:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012171720000000001 2023-10-20T03:36:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.473456+0000", 2023-10-20T03:36:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8139000000000002e-05 2023-10-20T03:36:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.473479+0000", 2023-10-20T03:36:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2446000000000002e-05 2023-10-20T03:36:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.473634+0000", 2023-10-20T03:36:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015524000000000001 2023-10-20T03:36:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.473657+0000", 2023-10-20T03:36:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3190999999999999e-05 2023-10-20T03:36:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.495030+0000", 2023-10-20T03:36:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021373098 2023-10-20T03:36:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:37.495123+0000", 2023-10-20T03:36:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2170999999999996e-05 2023-10-20T03:36:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/31 ECSubWrite(tid=246, reqid=client.4570.0:539, at_version=64'63, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:36:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.898724+0000", 2023-10-20T03:36:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.840996917999998, 2023-10-20T03:36:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030365014999999999, 2023-10-20T03:36:13.857 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898724+0000", 2023-10-20T03:36:13.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898724+0000", 2023-10-20T03:36:13.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898725+0000", 2023-10-20T03:36:13.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2023-10-20T03:36:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899207+0000", 2023-10-20T03:36:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048263100000000002 2023-10-20T03:36:13.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899243+0000", 2023-10-20T03:36:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5139999999999999e-05 2023-10-20T03:36:13.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899266+0000", 2023-10-20T03:36:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3142999999999998e-05 2023-10-20T03:36:13.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899450+0000", 2023-10-20T03:36:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018391999999999999 2023-10-20T03:36:13.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899477+0000", 2023-10-20T03:36:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7889e-05 2023-10-20T03:36:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.928985+0000", 2023-10-20T03:36:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029507230999999998 2023-10-20T03:36:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.929089+0000", 2023-10-20T03:36:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104615 2023-10-20T03:36:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.867 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/31 ECSubWrite(tid=247, reqid=client.4570.0:540, at_version=64'64, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:36:13.868 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.913105+0000", 2023-10-20T03:36:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.826616566999999, 2023-10-20T03:36:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028811000999999999, 2023-10-20T03:36:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.869 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913105+0000", 2023-10-20T03:36:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913105+0000", 2023-10-20T03:36:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913106+0000", 2023-10-20T03:36:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6530000000000001e-06 2023-10-20T03:36:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913264+0000", 2023-10-20T03:36:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015735400000000001 2023-10-20T03:36:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913297+0000", 2023-10-20T03:36:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3099000000000003e-05 2023-10-20T03:36:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913312+0000", 2023-10-20T03:36:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5713e-05 2023-10-20T03:36:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.916852+0000", 2023-10-20T03:36:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003540007 2023-10-20T03:36:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.916885+0000", 2023-10-20T03:36:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2716000000000003e-05 2023-10-20T03:36:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.941775+0000", 2023-10-20T03:36:13.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024890223999999999 2023-10-20T03:36:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.941916+0000", 2023-10-20T03:36:13.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140235 2023-10-20T03:36:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.879 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.3s6 70/56 ECSubWrite(tid=55, reqid=client.4570.0:556, at_version=70'40, trim_to=0'0, roll_forward_to=60'38))", 2023-10-20T03:36:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:51.966857+0000", 2023-10-20T03:36:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.772864125000002, 2023-10-20T03:36:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017176543999999998, 2023-10-20T03:36:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.880 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966857+0000", 2023-10-20T03:36:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966857+0000", 2023-10-20T03:36:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.966858+0000", 2023-10-20T03:36:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2699999999999999e-07 2023-10-20T03:36:13.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967036+0000", 2023-10-20T03:36:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017807700000000001 2023-10-20T03:36:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967064+0000", 2023-10-20T03:36:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8748999999999999e-05 2023-10-20T03:36:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967087+0000", 2023-10-20T03:36:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2240000000000001e-05 2023-10-20T03:36:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967329+0000", 2023-10-20T03:36:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024282899999999999 2023-10-20T03:36:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.967360+0000", 2023-10-20T03:36:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0094e-05 2023-10-20T03:36:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.983937+0000", 2023-10-20T03:36:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016577881999999999 2023-10-20T03:36:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:51.984034+0000", 2023-10-20T03:36:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6145999999999997e-05 2023-10-20T03:36:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es1 71/31 ECSubWrite(tid=52, reqid=client.4570.0:589, at_version=71'24, trim_to=0'0, roll_forward_to=61'23))", 2023-10-20T03:36:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:53.004322+0000", 2023-10-20T03:36:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.735399545, 2023-10-20T03:36:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044306533000000002, 2023-10-20T03:36:13.891 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004322+0000", 2023-10-20T03:36:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004322+0000", 2023-10-20T03:36:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004324+0000", 2023-10-20T03:36:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.187e-06 2023-10-20T03:36:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004869+0000", 2023-10-20T03:36:13.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054558099999999995 2023-10-20T03:36:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004900+0000", 2023-10-20T03:36:13.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0366999999999999e-05 2023-10-20T03:36:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004937+0000", 2023-10-20T03:36:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7509000000000003e-05 2023-10-20T03:36:13.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.005151+0000", 2023-10-20T03:36:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021337 2023-10-20T03:36:13.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.005177+0000", 2023-10-20T03:36:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6650000000000001e-05 2023-10-20T03:36:13.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.048582+0000", 2023-10-20T03:36:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043404953000000003 2023-10-20T03:36:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.048628+0000", 2023-10-20T03:36:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5915999999999999e-05 2023-10-20T03:36:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.901 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.3s6 71/56 ECSubWrite(tid=64, reqid=client.4570.0:576, at_version=71'47, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:13.902 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:53.007051+0000", 2023-10-20T03:36:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.732670683999999, 2023-10-20T03:36:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023764014999999999, 2023-10-20T03:36:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.903 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007051+0000", 2023-10-20T03:36:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007051+0000", 2023-10-20T03:36:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007053+0000", 2023-10-20T03:36:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.356e-06 2023-10-20T03:36:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007379+0000", 2023-10-20T03:36:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000326612 2023-10-20T03:36:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007419+0000", 2023-10-20T03:36:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9943e-05 2023-10-20T03:36:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007443+0000", 2023-10-20T03:36:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3436000000000001e-05 2023-10-20T03:36:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007617+0000", 2023-10-20T03:36:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000173804 2023-10-20T03:36:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.007643+0000", 2023-10-20T03:36:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6720000000000002e-05 2023-10-20T03:36:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.030749+0000", 2023-10-20T03:36:13.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023105173999999999 2023-10-20T03:36:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.030815+0000", 2023-10-20T03:36:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5969999999999993e-05 2023-10-20T03:36:13.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es1 71/31 ECSubWrite(tid=53, reqid=client.4570.0:590, at_version=71'25, trim_to=0'0, roll_forward_to=61'23))", 2023-10-20T03:36:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:53.014260+0000", 2023-10-20T03:36:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.725461071000002, 2023-10-20T03:36:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034398564999999999, 2023-10-20T03:36:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.914 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014260+0000", 2023-10-20T03:36:13.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014260+0000", 2023-10-20T03:36:13.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014262+0000", 2023-10-20T03:36:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8470000000000001e-06 2023-10-20T03:36:13.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014803+0000", 2023-10-20T03:36:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054058100000000005 2023-10-20T03:36:13.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014844+0000", 2023-10-20T03:36:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1383e-05 2023-10-20T03:36:13.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014867+0000", 2023-10-20T03:36:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3116e-05 2023-10-20T03:36:13.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.018959+0000", 2023-10-20T03:36:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040918090000000001 2023-10-20T03:36:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.018986+0000", 2023-10-20T03:36:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7353999999999999e-05 2023-10-20T03:36:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.048635+0000", 2023-10-20T03:36:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029648364 2023-10-20T03:36:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.048659+0000", 2023-10-20T03:36:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4111e-05 2023-10-20T03:36:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 77/31 ECSubWrite(tid=303, reqid=client.4570.0:632, at_version=77'73, trim_to=0'0, roll_forward_to=68'72))", 2023-10-20T03:36:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:58.785608+0000", 2023-10-20T03:36:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.954113251999999, 2023-10-20T03:36:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037712515000000002, 2023-10-20T03:36:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.785608+0000", 2023-10-20T03:36:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.785608+0000", 2023-10-20T03:36:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.785610+0000", 2023-10-20T03:36:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.72e-06 2023-10-20T03:36:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.788264+0000", 2023-10-20T03:36:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026540629999999999 2023-10-20T03:36:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.788295+0000", 2023-10-20T03:36:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1510000000000002e-05 2023-10-20T03:36:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.788311+0000", 2023-10-20T03:36:13.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5739999999999998e-05 2023-10-20T03:36:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.788487+0000", 2023-10-20T03:36:13.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176157 2023-10-20T03:36:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.788522+0000", 2023-10-20T03:36:13.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4700000000000003e-05 2023-10-20T03:36:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.823243+0000", 2023-10-20T03:36:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034721152999999998 2023-10-20T03:36:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.823320+0000", 2023-10-20T03:36:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7472000000000006e-05 2023-10-20T03:36:13.935 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.936 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:723 3.2s0 3:4f9c85d4:::smithi17039300-39:head [rmxattr [jVY`Wq[rUN`kr[k`eq^fOUOSk in=26b,rmxattr ^^[OYTcRcbV^X^YZXUd[aUhcdnNYtNgXUfqnaaVR in=40b,rmxattr _tchprYSfT_SpoXkYcVnYriaVUaom\\t_oaRn[PRT in=40b] snapc 33=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2023-10-20T03:36:13.937 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:09.078710+0000", 2023-10-20T03:36:13.937 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6610113110000002, 2023-10-20T03:36:13.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017145117000000001, 2023-10-20T03:36:13.937 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.937 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 723 2023-10-20T03:36:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.938 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.078710+0000", 2023-10-20T03:36:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.078710+0000", 2023-10-20T03:36:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.078713+0000", 2023-10-20T03:36:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1099999999999999e-06 2023-10-20T03:36:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.078730+0000", 2023-10-20T03:36:13.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7289e-05 2023-10-20T03:36:13.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.078768+0000", 2023-10-20T03:36:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7574999999999999e-05 2023-10-20T03:36:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.078786+0000", 2023-10-20T03:36:13.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7723999999999998e-05 2023-10-20T03:36:13.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.078932+0000", 2023-10-20T03:36:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014603199999999999 2023-10-20T03:36:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2023-10-20T03:36:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.078951+0000", 2023-10-20T03:36:13.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9383000000000001e-05 2023-10-20T03:36:13.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.086051+0000", 2023-10-20T03:36:13.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071002189999999996 2023-10-20T03:36:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.086378+0000", 2023-10-20T03:36:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032643100000000002 2023-10-20T03:36:13.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.088519+0000", 2023-10-20T03:36:13.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021409279999999998 2023-10-20T03:36:13.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.091370+0000", 2023-10-20T03:36:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028517080000000001 2023-10-20T03:36:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.095565+0000", 2023-10-20T03:36:13.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041945660000000003 2023-10-20T03:36:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:09.095855+0000", 2023-10-20T03:36:13.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029015200000000001 2023-10-20T03:36:13.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.953 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:781 3.2s0 3:4f9c85d4:::smithi17039300-39:head [append 0~0] snapc 39=[39,38,37,36] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2023-10-20T03:36:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:13.124299+0000", 2023-10-20T03:36:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.61542244300000004, 2023-10-20T03:36:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025289488999999998, 2023-10-20T03:36:13.954 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 781 2023-10-20T03:36:13.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.956 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124299+0000", 2023-10-20T03:36:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124299+0000", 2023-10-20T03:36:13.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124302+0000", 2023-10-20T03:36:13.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.117e-06 2023-10-20T03:36:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124314+0000", 2023-10-20T03:36:13.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1916000000000001e-05 2023-10-20T03:36:13.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124350+0000", 2023-10-20T03:36:13.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2023-10-20T03:36:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124367+0000", 2023-10-20T03:36:13.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7312999999999998e-05 2023-10-20T03:36:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124518+0000", 2023-10-20T03:36:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015137000000000001 2023-10-20T03:36:13.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2023-10-20T03:36:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124535+0000", 2023-10-20T03:36:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6929e-05 2023-10-20T03:36:13.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.132271+0000", 2023-10-20T03:36:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077359580000000002 2023-10-20T03:36:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.132552+0000", 2023-10-20T03:36:13.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028091200000000002 2023-10-20T03:36:13.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.135165+0000", 2023-10-20T03:36:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026132569999999999 2023-10-20T03:36:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.143149+0000", 2023-10-20T03:36:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079839249999999994 2023-10-20T03:36:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.149335+0000", 2023-10-20T03:36:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061859870000000004 2023-10-20T03:36:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.149588+0000", 2023-10-20T03:36:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025285599999999998 2023-10-20T03:36:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.970 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:782 3.2s0 3:4f9c85d4:::smithi17039300-39:head [rollback 37] snapc 39=[39,38,37,36] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2023-10-20T03:36:13.970 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:13.124412+0000", 2023-10-20T03:36:13.970 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.61530918499999998, 2023-10-20T03:36:13.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028725628, 2023-10-20T03:36:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.971 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 782 2023-10-20T03:36:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124412+0000", 2023-10-20T03:36:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124412+0000", 2023-10-20T03:36:13.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124413+0000", 2023-10-20T03:36:13.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5600000000000004e-07 2023-10-20T03:36:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124420+0000", 2023-10-20T03:36:13.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.323e-06 2023-10-20T03:36:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124436+0000", 2023-10-20T03:36:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5636000000000001e-05 2023-10-20T03:36:13.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124447+0000", 2023-10-20T03:36:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0807e-05 2023-10-20T03:36:13.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124745+0000", 2023-10-20T03:36:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029878899999999999 2023-10-20T03:36:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:13.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124777+0000", 2023-10-20T03:36:13.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1322000000000003e-05 2023-10-20T03:36:13.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.136698+0000", 2023-10-20T03:36:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011920984000000001 2023-10-20T03:36:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.137001+0000", 2023-10-20T03:36:13.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030286499999999998 2023-10-20T03:36:13.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.138685+0000", 2023-10-20T03:36:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016848570000000001 2023-10-20T03:36:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.144644+0000", 2023-10-20T03:36:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059585619999999997 2023-10-20T03:36:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:13.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.152961+0000", 2023-10-20T03:36:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0083170210000000008 2023-10-20T03:36:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:13.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.153138+0000", 2023-10-20T03:36:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176606 2023-10-20T03:36:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.986 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:783 3.2s0 3:4f9c85d4:::smithi17039300-39:head [append 0~0] snapc 39=[39,38,37,36] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2023-10-20T03:36:13.986 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:13.124487+0000", 2023-10-20T03:36:13.986 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.61523436099999995, 2023-10-20T03:36:13.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030469586, 2023-10-20T03:36:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:13.987 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 783 2023-10-20T03:36:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124487+0000", 2023-10-20T03:36:13.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:13.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124487+0000", 2023-10-20T03:36:13.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:13.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:13.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124488+0000", 2023-10-20T03:36:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0999999999999998e-07 2023-10-20T03:36:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:13.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124492+0000", 2023-10-20T03:36:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1969999999999998e-06 2023-10-20T03:36:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124504+0000", 2023-10-20T03:36:13.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1956e-05 2023-10-20T03:36:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124512+0000", 2023-10-20T03:36:13.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.106e-06 2023-10-20T03:36:13.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124846+0000", 2023-10-20T03:36:13.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000334445 2023-10-20T03:36:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124859+0000", 2023-10-20T03:36:13.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3179999999999999e-05 2023-10-20T03:36:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.140289+0000", 2023-10-20T03:36:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015429506000000001 2023-10-20T03:36:13.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.140617+0000", 2023-10-20T03:36:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000327762 2023-10-20T03:36:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.142313+0000", 2023-10-20T03:36:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001696576 2023-10-20T03:36:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.147591+0000", 2023-10-20T03:36:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052780689999999998 2023-10-20T03:36:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:13.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.154689+0000", 2023-10-20T03:36:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070977499999999999 2023-10-20T03:36:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:14.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.154956+0000", 2023-10-20T03:36:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026732900000000001 2023-10-20T03:36:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:14.001 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:14.002 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:36:14.003 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:36:13.740+0000 7ff0bcb75640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35548) UID: 0 2023-10-20T03:36:14.004 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-10-20T03:36:14.004 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:14.000+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:14.105 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:14.100+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:14.134 INFO:tasks.rados.rados.0.smithi170.stdout:322: done (5 left) 2023-10-20T03:36:14.135 INFO:tasks.rados.rados.0.smithi170.stdout:323: read oid 35 snap 28 2023-10-20T03:36:14.135 INFO:tasks.rados.rados.0.smithi170.stdout:323: expect (ObjNum 123 snap 25 seq_num 123) 2023-10-20T03:36:14.135 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 20 current snap is 30 2023-10-20T03:36:14.135 INFO:tasks.rados.rados.0.smithi170.stdout:324: seq_num 136 ranges {2703360=737280,3440640=770048,4210688=540672,4751360=245760} 2023-10-20T03:36:14.136 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:14.136 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:36:14.136 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2023-10-20T03:36:14.136 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:14.138 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-10-20T03:36:14.159 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-10-20T03:36:14.160 INFO:tasks.rados.rados.0.smithi170.stdout:324: writing smithi17039300-20 from 2703360 to 3440640 tid 1 2023-10-20T03:36:14.164 INFO:tasks.rados.rados.0.smithi170.stdout:324: writing smithi17039300-20 from 3440640 to 4210688 tid 2 2023-10-20T03:36:14.168 INFO:tasks.rados.rados.0.smithi170.stdout:324: writing smithi17039300-20 from 4210688 to 4751360 tid 3 2023-10-20T03:36:14.170 INFO:tasks.rados.rados.0.smithi170.stdout:324: writing smithi17039300-20 from 4751360 to 4997120 tid 4 2023-10-20T03:36:14.170 INFO:tasks.rados.rados.0.smithi170.stdout:325: read oid 12 snap -1 2023-10-20T03:36:14.170 INFO:tasks.rados.rados.0.smithi170.stdout:325: expect deleted 2023-10-20T03:36:14.171 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 10 current snap is 30 2023-10-20T03:36:14.171 INFO:tasks.rados.rados.0.smithi170.stdout:320: finishing rollback tid 0 to smithi17039300-39 2023-10-20T03:36:14.171 INFO:tasks.rados.rados.0.smithi170.stdout:320: finishing rollback tid 1 to smithi17039300-39 2023-10-20T03:36:14.171 INFO:tasks.rados.rados.0.smithi170.stdout:318: finishing rollback tid 0 to smithi17039300-23 2023-10-20T03:36:14.171 INFO:tasks.rados.rados.0.smithi170.stdout:318: finishing rollback tid 1 to smithi17039300-23 2023-10-20T03:36:14.171 INFO:tasks.rados.rados.0.smithi170.stdout:320: finishing rollback tid 2 to smithi17039300-39 2023-10-20T03:36:14.172 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 73 (ObjNum 1185 snap 0 seq_num 1929370824) dirty exists 2023-10-20T03:36:14.172 INFO:tasks.rados.rados.0.smithi170.stdout:318: finishing rollback tid 2 to smithi17039300-23 2023-10-20T03:36:14.172 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 23 v 77 (ObjNum 77 snap 5 seq_num 77) dirty exists 2023-10-20T03:36:14.172 INFO:tasks.rados.rados.0.smithi170.stdout:319: expect (ObjNum 100 snap 13 seq_num 100) 2023-10-20T03:36:14.175 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:36:14.205 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:14.201+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:14.261 INFO:tasks.rados.rados.0.smithi170.stdout:321: finishing copy_from to smithi17039300-48 2023-10-20T03:36:14.261 INFO:tasks.rados.rados.0.smithi170.stdout:321: got expected ENOENT (src dne) 2023-10-20T03:36:14.261 INFO:tasks.rados.rados.0.smithi170.stdout:321: finishing copy_from racing read to smithi17039300-48 2023-10-20T03:36:14.262 INFO:tasks.rados.rados.0.smithi170.stdout:315: expect (ObjNum 2 snap 0 seq_num 2) 2023-10-20T03:36:14.263 INFO:tasks.rados.rados.0.smithi170.stdout:323: expect (ObjNum 123 snap 25 seq_num 123) 2023-10-20T03:36:14.282 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:14.283 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:36:14.283 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:36:14.283 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:36:14.283 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:14.297 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-10-20T03:36:14.306 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:14.304+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:14.335 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:36:14.335 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.341 INFO:tasks.rados.rados.0.smithi170.stdout:326: seq_num 137 ranges {3178496=753664,3932160=786432,4718592=655360,5373952=114688} 2023-10-20T03:36:14.355 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:36:14.355 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.362 INFO:tasks.rados.rados.0.smithi170.stdout:326: writing smithi17039300-10 from 3178496 to 3932160 tid 1 2023-10-20T03:36:14.366 INFO:tasks.rados.rados.0.smithi170.stdout:326: writing smithi17039300-10 from 3932160 to 4718592 tid 2 2023-10-20T03:36:14.371 INFO:tasks.rados.rados.0.smithi170.stdout:326: writing smithi17039300-10 from 4718592 to 5373952 tid 3 2023-10-20T03:36:14.373 INFO:tasks.rados.rados.0.smithi170.stdout:326: writing smithi17039300-10 from 5373952 to 5488640 tid 4 2023-10-20T03:36:14.373 INFO:tasks.rados.rados.0.smithi170.stdout:315: done (8 left) 2023-10-20T03:36:14.373 INFO:tasks.rados.rados.0.smithi170.stdout:318: done (7 left) 2023-10-20T03:36:14.374 INFO:tasks.rados.rados.0.smithi170.stdout:319: done (6 left) 2023-10-20T03:36:14.374 INFO:tasks.rados.rados.0.smithi170.stdout:320: done (5 left) 2023-10-20T03:36:14.374 INFO:tasks.rados.rados.0.smithi170.stdout:321: done (4 left) 2023-10-20T03:36:14.374 INFO:tasks.rados.rados.0.smithi170.stdout:323: done (3 left) 2023-10-20T03:36:14.375 INFO:tasks.rados.rados.0.smithi170.stdout:327: setattr oid 49 current snap is 30 2023-10-20T03:36:14.376 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:36:14.376 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.377 INFO:tasks.rados.rados.0.smithi170.stdout:328: copy_from oid 5 from oid 35 current snap is 30 2023-10-20T03:36:14.377 INFO:tasks.rados.rados.0.smithi170.stdout:329: delete oid 32 current snap is 30 2023-10-20T03:36:14.395 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:36:14.395 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.406 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:14.401+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:14.415 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:36:14.416 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.417 INFO:tasks.rados.rados.0.smithi170.stdout:326: finishing write tid 1 to smithi17039300-10 2023-10-20T03:36:14.418 INFO:tasks.rados.rados.0.smithi170.stdout:326: finishing write tid 2 to smithi17039300-10 2023-10-20T03:36:14.427 INFO:tasks.rados.rados.0.smithi170.stdout:326: finishing write tid 3 to smithi17039300-10 2023-10-20T03:36:14.428 INFO:tasks.rados.rados.0.smithi170.stdout:326: finishing write tid 4 to smithi17039300-10 2023-10-20T03:36:14.429 INFO:tasks.rados.rados.0.smithi170.stdout:326: finishing write tid 5 to smithi17039300-10 2023-10-20T03:36:14.433 INFO:tasks.rados.rados.0.smithi170.stdout:326: finishing write tid 6 to smithi17039300-10 2023-10-20T03:36:14.433 INFO:tasks.rados.rados.0.smithi170.stdout:326: oid 10 updating version 0 to 72 2023-10-20T03:36:14.434 INFO:tasks.rados.rados.0.smithi170.stdout:326: oid 10 updating version 72 to 74 2023-10-20T03:36:14.434 INFO:tasks.rados.rados.0.smithi170.stdout:326: oid 10 version 74 is already newer than 73 2023-10-20T03:36:14.434 INFO:tasks.rados.rados.0.smithi170.stdout:326: oid 10 version 74 is already newer than 71 2023-10-20T03:36:14.434 INFO:tasks.rados.rados.0.smithi170.stdout:326: oid 10 updating version 74 to 75 2023-10-20T03:36:14.435 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 10 v 75 (ObjNum 137 snap 30 seq_num 137) dirty exists 2023-10-20T03:36:14.435 INFO:tasks.rados.rados.0.smithi170.stdout:326: left oid 10 (ObjNum 137 snap 30 seq_num 137) 2023-10-20T03:36:14.437 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:36:14.437 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.457 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2023-10-20T03:36:14.457 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.463 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:14.463 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:14.476 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:36:14.477 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.478 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-10-20T03:36:14.493 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:36:14.493 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.508 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:14.503+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:14.513 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9: {} 2023-10-20T03:36:14.513 INFO:teuthology.orchestra.run.smithi033.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.535 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:36:14.535 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.553 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:36:14.554 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.572 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:36:14.572 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.594 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:36:14.595 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.608 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:14.605+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:14.616 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:36:14.612+0000 7f2a57fff640 0 -- 172.21.15.33:0/1017826523 >> [v2:172.21.15.64:6816/2940007286,v1:172.21.15.64:6817/2940007286] conn(0x7f2a6021f4b0 msgr2=0x7f2a602218c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-10-20T03:36:14.620 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:36:14.643 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:14.643 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:14.644 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:36:14.644 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:36:14.658 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-10-20T03:36:14.709 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:14.704+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:14.810 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:14.805+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:14.822 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:14.822 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:14.836 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-10-20T03:36:14.911 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:36:14.907+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:36:15.004 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:15.004 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:15.013 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:15.011+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:15.018 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-10-20T03:36:15.114 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:15.111+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:15.130 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2023-10-20T03:36:15.147 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:15.148 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:15.168 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 49 v 67 (ObjNum 91 snap 11 seq_num 91) dirty exists 2023-10-20T03:36:15.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-10-20T03:36:15.187 INFO:tasks.rados.rados.0.smithi170.stdout:325: done (5 left) 2023-10-20T03:36:15.188 INFO:tasks.rados.rados.0.smithi170.stdout:326: done (4 left) 2023-10-20T03:36:15.188 INFO:tasks.rados.rados.0.smithi170.stdout:327: done (3 left) 2023-10-20T03:36:15.188 INFO:tasks.rados.rados.0.smithi170.stdout:329: done (2 left) 2023-10-20T03:36:15.188 INFO:tasks.rados.rados.0.smithi170.stdout:330: read oid 49 snap 29 2023-10-20T03:36:15.189 INFO:tasks.rados.rados.0.smithi170.stdout:330: expect (ObjNum 91 snap 11 seq_num 91) 2023-10-20T03:36:15.189 INFO:tasks.rados.rados.0.smithi170.stdout:331: setattr oid 44 current snap is 30 2023-10-20T03:36:15.189 INFO:tasks.rados.rados.0.smithi170.stdout:332: read oid 3 snap -1 2023-10-20T03:36:15.189 INFO:tasks.rados.rados.0.smithi170.stdout:332: expect (ObjNum 2 snap 0 seq_num 2) 2023-10-20T03:36:15.190 INFO:tasks.rados.rados.0.smithi170.stdout:333: snap_create 2023-10-20T03:36:15.214 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:15.212+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:15.227 INFO:tasks.rados.rados.0.smithi170.stdout:330: expect (ObjNum 91 snap 11 seq_num 91) 2023-10-20T03:36:15.262 INFO:tasks.ceph.osd.9.smithi040.stderr:./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7ff09d91b640 time 2023-10-20T03:36:15.262613+0000 2023-10-20T03:36:15.262 INFO:tasks.ceph.osd.9.smithi040.stderr:./src/osd/SnapMapper.cc: 620: FAILED ceph_assert(out->size() == 0) 2023-10-20T03:36:15.270 INFO:tasks.ceph.osd.9.smithi040.stderr: ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 2023-10-20T03:36:15.270 INFO:tasks.ceph.osd.9.smithi040.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x118) [0x56537327c4a1] 2023-10-20T03:36:15.270 INFO:tasks.ceph.osd.9.smithi040.stderr: 2: ceph-osd(+0xcd4658) [0x56537327c658] 2023-10-20T03:36:15.271 INFO:tasks.ceph.osd.9.smithi040.stderr: 3: ceph-osd(+0xc8fd62) [0x565373237d62] 2023-10-20T03:36:15.271 INFO:tasks.ceph.osd.9.smithi040.stderr: 4: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x5653734cb4a2] 2023-10-20T03:36:15.271 INFO:tasks.ceph.osd.9.smithi040.stderr: 5: ceph-osd(+0xd7e9c1) [0x5653733269c1] 2023-10-20T03:36:15.271 INFO:tasks.ceph.osd.9.smithi040.stderr: 6: ceph-osd(+0x1695efb) [0x565373c3defb] 2023-10-20T03:36:15.271 INFO:tasks.ceph.osd.9.smithi040.stderr: 7: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x5653734683d8] 2023-10-20T03:36:15.271 INFO:tasks.ceph.osd.9.smithi040.stderr: 8: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x56537359dfbf] 2023-10-20T03:36:15.272 INFO:tasks.ceph.osd.9.smithi040.stderr: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5653733bcfa3] 2023-10-20T03:36:15.272 INFO:tasks.ceph.osd.9.smithi040.stderr: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x56537386d923] 2023-10-20T03:36:15.272 INFO:tasks.ceph.osd.9.smithi040.stderr: 11: ceph-osd(+0x12c5e54) [0x56537386de54] 2023-10-20T03:36:15.272 INFO:tasks.ceph.osd.9.smithi040.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff0c07d3b43] 2023-10-20T03:36:15.272 INFO:tasks.ceph.osd.9.smithi040.stderr: 13: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff0c0865a00] 2023-10-20T03:36:15.272 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:36:15.264+0000 7ff09d91b640 -1 ./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7ff09d91b640 time 2023-10-20T03:36:15.262613+0000 2023-10-20T03:36:15.273 INFO:tasks.ceph.osd.9.smithi040.stderr:./src/osd/SnapMapper.cc: 620: FAILED ceph_assert(out->size() == 0) 2023-10-20T03:36:15.273 INFO:tasks.ceph.osd.9.smithi040.stderr: 2023-10-20T03:36:15.273 INFO:tasks.ceph.osd.9.smithi040.stderr: ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 2023-10-20T03:36:15.273 INFO:tasks.ceph.osd.9.smithi040.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x118) [0x56537327c4a1] 2023-10-20T03:36:15.273 INFO:tasks.ceph.osd.9.smithi040.stderr: 2: ceph-osd(+0xcd4658) [0x56537327c658] 2023-10-20T03:36:15.274 INFO:tasks.ceph.osd.9.smithi040.stderr: 3: ceph-osd(+0xc8fd62) [0x565373237d62] 2023-10-20T03:36:15.274 INFO:tasks.ceph.osd.9.smithi040.stderr: 4: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x5653734cb4a2] 2023-10-20T03:36:15.274 INFO:tasks.ceph.osd.9.smithi040.stderr: 5: ceph-osd(+0xd7e9c1) [0x5653733269c1] 2023-10-20T03:36:15.274 INFO:tasks.ceph.osd.9.smithi040.stderr: 6: ceph-osd(+0x1695efb) [0x565373c3defb] 2023-10-20T03:36:15.274 INFO:tasks.ceph.osd.9.smithi040.stderr: 7: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x5653734683d8] 2023-10-20T03:36:15.274 INFO:tasks.ceph.osd.9.smithi040.stderr: 8: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x56537359dfbf] 2023-10-20T03:36:15.275 INFO:tasks.ceph.osd.9.smithi040.stderr: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5653733bcfa3] 2023-10-20T03:36:15.275 INFO:tasks.ceph.osd.9.smithi040.stderr: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x56537386d923] 2023-10-20T03:36:15.275 INFO:tasks.ceph.osd.9.smithi040.stderr: 11: ceph-osd(+0x12c5e54) [0x56537386de54] 2023-10-20T03:36:15.275 INFO:tasks.ceph.osd.9.smithi040.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff0c07d3b43] 2023-10-20T03:36:15.275 INFO:tasks.ceph.osd.9.smithi040.stderr: 13: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff0c0865a00] 2023-10-20T03:36:15.276 INFO:tasks.ceph.osd.9.smithi040.stderr: 2023-10-20T03:36:15.276 INFO:tasks.ceph.osd.9.smithi040.stderr:*** Caught signal (Aborted) ** 2023-10-20T03:36:15.276 INFO:tasks.ceph.osd.9.smithi040.stderr: in thread 7ff09d91b640 thread_name:tp_osd_tp 2023-10-20T03:36:15.277 INFO:tasks.ceph.osd.9.smithi040.stderr: ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 2023-10-20T03:36:15.277 INFO:tasks.ceph.osd.9.smithi040.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff0c0781520] 2023-10-20T03:36:15.277 INFO:tasks.ceph.osd.9.smithi040.stderr: 2: pthread_kill() 2023-10-20T03:36:15.277 INFO:tasks.ceph.osd.9.smithi040.stderr: 3: raise() 2023-10-20T03:36:15.278 INFO:tasks.ceph.osd.9.smithi040.stderr: 4: abort() 2023-10-20T03:36:15.278 INFO:tasks.ceph.osd.9.smithi040.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x56537327c4f7] 2023-10-20T03:36:15.278 INFO:tasks.ceph.osd.9.smithi040.stderr: 6: ceph-osd(+0xcd4658) [0x56537327c658] 2023-10-20T03:36:15.279 INFO:tasks.ceph.osd.9.smithi040.stderr: 7: ceph-osd(+0xc8fd62) [0x565373237d62] 2023-10-20T03:36:15.279 INFO:tasks.ceph.osd.9.smithi040.stderr: 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x5653734cb4a2] 2023-10-20T03:36:15.279 INFO:tasks.ceph.osd.9.smithi040.stderr: 9: ceph-osd(+0xd7e9c1) [0x5653733269c1] 2023-10-20T03:36:15.279 INFO:tasks.ceph.osd.9.smithi040.stderr: 10: ceph-osd(+0x1695efb) [0x565373c3defb] 2023-10-20T03:36:15.279 INFO:tasks.ceph.osd.9.smithi040.stderr: 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x5653734683d8] 2023-10-20T03:36:15.280 INFO:tasks.ceph.osd.9.smithi040.stderr: 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x56537359dfbf] 2023-10-20T03:36:15.280 INFO:tasks.ceph.osd.9.smithi040.stderr: 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5653733bcfa3] 2023-10-20T03:36:15.280 INFO:tasks.ceph.osd.9.smithi040.stderr: 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x56537386d923] 2023-10-20T03:36:15.280 INFO:tasks.ceph.osd.9.smithi040.stderr: 15: ceph-osd(+0x12c5e54) [0x56537386de54] 2023-10-20T03:36:15.280 INFO:tasks.ceph.osd.9.smithi040.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff0c07d3b43] 2023-10-20T03:36:15.281 INFO:tasks.ceph.osd.9.smithi040.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff0c0865a00] 2023-10-20T03:36:15.281 INFO:tasks.ceph.osd.9.smithi040.stderr:2023-10-20T03:36:15.272+0000 7ff09d91b640 -1 *** Caught signal (Aborted) ** 2023-10-20T03:36:15.281 INFO:tasks.ceph.osd.9.smithi040.stderr: in thread 7ff09d91b640 thread_name:tp_osd_tp 2023-10-20T03:36:15.281 INFO:tasks.ceph.osd.9.smithi040.stderr: 2023-10-20T03:36:15.282 INFO:tasks.ceph.osd.9.smithi040.stderr: ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 2023-10-20T03:36:15.282 INFO:tasks.ceph.osd.9.smithi040.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff0c0781520] 2023-10-20T03:36:15.282 INFO:tasks.ceph.osd.9.smithi040.stderr: 2: pthread_kill() 2023-10-20T03:36:15.282 INFO:tasks.ceph.osd.9.smithi040.stderr: 3: raise() 2023-10-20T03:36:15.282 INFO:tasks.ceph.osd.9.smithi040.stderr: 4: abort() 2023-10-20T03:36:15.283 INFO:tasks.ceph.osd.9.smithi040.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x56537327c4f7] 2023-10-20T03:36:15.283 INFO:tasks.ceph.osd.9.smithi040.stderr: 6: ceph-osd(+0xcd4658) [0x56537327c658] 2023-10-20T03:36:15.283 INFO:tasks.ceph.osd.9.smithi040.stderr: 7: ceph-osd(+0xc8fd62) [0x565373237d62] 2023-10-20T03:36:15.283 INFO:tasks.ceph.osd.9.smithi040.stderr: 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x5653734cb4a2] 2023-10-20T03:36:15.283 INFO:tasks.ceph.osd.9.smithi040.stderr: 9: ceph-osd(+0xd7e9c1) [0x5653733269c1] 2023-10-20T03:36:15.284 INFO:tasks.ceph.osd.9.smithi040.stderr: 10: ceph-osd(+0x1695efb) [0x565373c3defb] 2023-10-20T03:36:15.284 INFO:tasks.ceph.osd.9.smithi040.stderr: 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x5653734683d8] 2023-10-20T03:36:15.284 INFO:tasks.ceph.osd.9.smithi040.stderr: 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x56537359dfbf] 2023-10-20T03:36:15.284 INFO:tasks.ceph.osd.9.smithi040.stderr: 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5653733bcfa3] 2023-10-20T03:36:15.284 INFO:tasks.ceph.osd.9.smithi040.stderr: 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x56537386d923] 2023-10-20T03:36:15.285 INFO:tasks.ceph.osd.9.smithi040.stderr: 15: ceph-osd(+0x12c5e54) [0x56537386de54] 2023-10-20T03:36:15.285 INFO:tasks.ceph.osd.9.smithi040.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff0c07d3b43] 2023-10-20T03:36:15.285 INFO:tasks.ceph.osd.9.smithi040.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff0c0865a00] 2023-10-20T03:36:15.285 INFO:tasks.ceph.osd.9.smithi040.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-20T03:36:15.285 INFO:tasks.ceph.osd.9.smithi040.stderr: 2023-10-20T03:36:15.298 INFO:tasks.ceph.osd.9.smithi040.stderr: -387> 2023-10-20T03:36:15.264+0000 7ff09d91b640 -1 ./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7ff09d91b640 time 2023-10-20T03:36:15.262613+0000 2023-10-20T03:36:15.298 INFO:tasks.ceph.osd.9.smithi040.stderr:./src/osd/SnapMapper.cc: 620: FAILED ceph_assert(out->size() == 0) 2023-10-20T03:36:15.298 INFO:tasks.ceph.osd.9.smithi040.stderr: 2023-10-20T03:36:15.298 INFO:tasks.ceph.osd.9.smithi040.stderr: ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 2023-10-20T03:36:15.299 INFO:tasks.ceph.osd.9.smithi040.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x118) [0x56537327c4a1] 2023-10-20T03:36:15.299 INFO:tasks.ceph.osd.9.smithi040.stderr: 2: ceph-osd(+0xcd4658) [0x56537327c658] 2023-10-20T03:36:15.299 INFO:tasks.ceph.osd.9.smithi040.stderr: 3: ceph-osd(+0xc8fd62) [0x565373237d62] 2023-10-20T03:36:15.299 INFO:tasks.ceph.osd.9.smithi040.stderr: 4: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x5653734cb4a2] 2023-10-20T03:36:15.299 INFO:tasks.ceph.osd.9.smithi040.stderr: 5: ceph-osd(+0xd7e9c1) [0x5653733269c1] 2023-10-20T03:36:15.299 INFO:tasks.ceph.osd.9.smithi040.stderr: 6: ceph-osd(+0x1695efb) [0x565373c3defb] 2023-10-20T03:36:15.300 INFO:tasks.ceph.osd.9.smithi040.stderr: 7: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x5653734683d8] 2023-10-20T03:36:15.300 INFO:tasks.ceph.osd.9.smithi040.stderr: 8: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x56537359dfbf] 2023-10-20T03:36:15.300 INFO:tasks.ceph.osd.9.smithi040.stderr: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5653733bcfa3] 2023-10-20T03:36:15.300 INFO:tasks.ceph.osd.9.smithi040.stderr: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x56537386d923] 2023-10-20T03:36:15.300 INFO:tasks.ceph.osd.9.smithi040.stderr: 11: ceph-osd(+0x12c5e54) [0x56537386de54] 2023-10-20T03:36:15.301 INFO:tasks.ceph.osd.9.smithi040.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff0c07d3b43] 2023-10-20T03:36:15.301 INFO:tasks.ceph.osd.9.smithi040.stderr: 13: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff0c0865a00] 2023-10-20T03:36:15.301 INFO:tasks.ceph.osd.9.smithi040.stderr: 2023-10-20T03:36:15.301 INFO:tasks.ceph.osd.9.smithi040.stderr: 0> 2023-10-20T03:36:15.272+0000 7ff09d91b640 -1 *** Caught signal (Aborted) ** 2023-10-20T03:36:15.301 INFO:tasks.ceph.osd.9.smithi040.stderr: in thread 7ff09d91b640 thread_name:tp_osd_tp 2023-10-20T03:36:15.301 INFO:tasks.ceph.osd.9.smithi040.stderr: 2023-10-20T03:36:15.302 INFO:tasks.ceph.osd.9.smithi040.stderr: ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 2023-10-20T03:36:15.302 INFO:tasks.ceph.osd.9.smithi040.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff0c0781520] 2023-10-20T03:36:15.302 INFO:tasks.ceph.osd.9.smithi040.stderr: 2: pthread_kill() 2023-10-20T03:36:15.302 INFO:tasks.ceph.osd.9.smithi040.stderr: 3: raise() 2023-10-20T03:36:15.302 INFO:tasks.ceph.osd.9.smithi040.stderr: 4: abort() 2023-10-20T03:36:15.302 INFO:tasks.ceph.osd.9.smithi040.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x56537327c4f7] 2023-10-20T03:36:15.303 INFO:tasks.ceph.osd.9.smithi040.stderr: 6: ceph-osd(+0xcd4658) [0x56537327c658] 2023-10-20T03:36:15.303 INFO:tasks.ceph.osd.9.smithi040.stderr: 7: ceph-osd(+0xc8fd62) [0x565373237d62] 2023-10-20T03:36:15.303 INFO:tasks.ceph.osd.9.smithi040.stderr: 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x5653734cb4a2] 2023-10-20T03:36:15.303 INFO:tasks.ceph.osd.9.smithi040.stderr: 9: ceph-osd(+0xd7e9c1) [0x5653733269c1] 2023-10-20T03:36:15.303 INFO:tasks.ceph.osd.9.smithi040.stderr: 10: ceph-osd(+0x1695efb) [0x565373c3defb] 2023-10-20T03:36:15.303 INFO:tasks.ceph.osd.9.smithi040.stderr: 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x5653734683d8] 2023-10-20T03:36:15.304 INFO:tasks.ceph.osd.9.smithi040.stderr: 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x56537359dfbf] 2023-10-20T03:36:15.304 INFO:tasks.ceph.osd.9.smithi040.stderr: 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5653733bcfa3] 2023-10-20T03:36:15.304 INFO:tasks.ceph.osd.9.smithi040.stderr: 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x56537386d923] 2023-10-20T03:36:15.304 INFO:tasks.ceph.osd.9.smithi040.stderr: 15: ceph-osd(+0x12c5e54) [0x56537386de54] 2023-10-20T03:36:15.304 INFO:tasks.ceph.osd.9.smithi040.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff0c07d3b43] 2023-10-20T03:36:15.304 INFO:tasks.ceph.osd.9.smithi040.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff0c0865a00] 2023-10-20T03:36:15.305 INFO:tasks.ceph.osd.9.smithi040.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-20T03:36:15.305 INFO:tasks.ceph.osd.9.smithi040.stderr: 2023-10-20T03:36:15.312 INFO:tasks.ceph.osd.9.smithi040.stderr: -1211> 2023-10-20T03:36:15.264+0000 7ff09d91b640 -1 ./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7ff09d91b640 time 2023-10-20T03:36:15.262613+0000 2023-10-20T03:36:15.312 INFO:tasks.ceph.osd.9.smithi040.stderr:./src/osd/SnapMapper.cc: 620: FAILED ceph_assert(out->size() == 0) 2023-10-20T03:36:15.312 INFO:tasks.ceph.osd.9.smithi040.stderr: 2023-10-20T03:36:15.313 INFO:tasks.ceph.osd.9.smithi040.stderr: ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 2023-10-20T03:36:15.313 INFO:tasks.ceph.osd.9.smithi040.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x118) [0x56537327c4a1] 2023-10-20T03:36:15.313 INFO:tasks.ceph.osd.9.smithi040.stderr: 2: ceph-osd(+0xcd4658) [0x56537327c658] 2023-10-20T03:36:15.313 INFO:tasks.ceph.osd.9.smithi040.stderr: 3: ceph-osd(+0xc8fd62) [0x565373237d62] 2023-10-20T03:36:15.313 INFO:tasks.ceph.osd.9.smithi040.stderr: 4: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x5653734cb4a2] 2023-10-20T03:36:15.314 INFO:tasks.ceph.osd.9.smithi040.stderr: 5: ceph-osd(+0xd7e9c1) [0x5653733269c1] 2023-10-20T03:36:15.314 INFO:tasks.ceph.osd.9.smithi040.stderr: 6: ceph-osd(+0x1695efb) [0x565373c3defb] 2023-10-20T03:36:15.314 INFO:tasks.ceph.osd.9.smithi040.stderr: 7: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x5653734683d8] 2023-10-20T03:36:15.314 INFO:tasks.ceph.osd.9.smithi040.stderr: 8: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x56537359dfbf] 2023-10-20T03:36:15.315 INFO:tasks.ceph.osd.9.smithi040.stderr: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5653733bcfa3] 2023-10-20T03:36:15.315 INFO:tasks.ceph.osd.9.smithi040.stderr: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x56537386d923] 2023-10-20T03:36:15.315 INFO:tasks.ceph.osd.9.smithi040.stderr: 11: ceph-osd(+0x12c5e54) [0x56537386de54] 2023-10-20T03:36:15.315 INFO:tasks.ceph.osd.9.smithi040.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff0c07d3b43] 2023-10-20T03:36:15.315 INFO:tasks.ceph.osd.9.smithi040.stderr: 13: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff0c0865a00] 2023-10-20T03:36:15.316 INFO:tasks.ceph.osd.9.smithi040.stderr: 2023-10-20T03:36:15.316 INFO:tasks.ceph.osd.9.smithi040.stderr: -824> 2023-10-20T03:36:15.272+0000 7ff09d91b640 -1 *** Caught signal (Aborted) ** 2023-10-20T03:36:15.316 INFO:tasks.ceph.osd.9.smithi040.stderr: in thread 7ff09d91b640 thread_name:tp_osd_tp 2023-10-20T03:36:15.316 INFO:tasks.ceph.osd.9.smithi040.stderr: 2023-10-20T03:36:15.316 INFO:tasks.ceph.osd.9.smithi040.stderr: ceph version 18.0.0-6703-g8a5df55d (8a5df55d22d3638066184c241442ff00fdc0ebdd) reef (dev) 2023-10-20T03:36:15.316 INFO:tasks.ceph.osd.9.smithi040.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff0c0781520] 2023-10-20T03:36:15.317 INFO:tasks.ceph.osd.9.smithi040.stderr: 2: pthread_kill() 2023-10-20T03:36:15.317 INFO:tasks.ceph.osd.9.smithi040.stderr: 3: raise() 2023-10-20T03:36:15.317 INFO:tasks.ceph.osd.9.smithi040.stderr: 4: abort() 2023-10-20T03:36:15.317 INFO:tasks.ceph.osd.9.smithi040.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x16e) [0x56537327c4f7] 2023-10-20T03:36:15.317 INFO:tasks.ceph.osd.9.smithi040.stderr: 6: ceph-osd(+0xcd4658) [0x56537327c658] 2023-10-20T03:36:15.317 INFO:tasks.ceph.osd.9.smithi040.stderr: 7: ceph-osd(+0xc8fd62) [0x565373237d62] 2023-10-20T03:36:15.318 INFO:tasks.ceph.osd.9.smithi040.stderr: 8: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x5653734cb4a2] 2023-10-20T03:36:15.318 INFO:tasks.ceph.osd.9.smithi040.stderr: 9: ceph-osd(+0xd7e9c1) [0x5653733269c1] 2023-10-20T03:36:15.318 INFO:tasks.ceph.osd.9.smithi040.stderr: 10: ceph-osd(+0x1695efb) [0x565373c3defb] 2023-10-20T03:36:15.318 INFO:tasks.ceph.osd.9.smithi040.stderr: 11: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x5653734683d8] 2023-10-20T03:36:15.318 INFO:tasks.ceph.osd.9.smithi040.stderr: 12: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x56537359dfbf] 2023-10-20T03:36:15.319 INFO:tasks.ceph.osd.9.smithi040.stderr: 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x5653733bcfa3] 2023-10-20T03:36:15.319 INFO:tasks.ceph.osd.9.smithi040.stderr: 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x56537386d923] 2023-10-20T03:36:15.319 INFO:tasks.ceph.osd.9.smithi040.stderr: 15: ceph-osd(+0x12c5e54) [0x56537386de54] 2023-10-20T03:36:15.319 INFO:tasks.ceph.osd.9.smithi040.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7ff0c07d3b43] 2023-10-20T03:36:15.319 INFO:tasks.ceph.osd.9.smithi040.stderr: 17: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7ff0c0865a00] 2023-10-20T03:36:15.320 INFO:tasks.ceph.osd.9.smithi040.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-10-20T03:36:15.320 INFO:tasks.ceph.osd.9.smithi040.stderr: 2023-10-20T03:36:15.321 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:15.316+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:15.335 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 44 v 77 (ObjNum 43 snap 0 seq_num 43) dirty exists 2023-10-20T03:36:15.335 INFO:tasks.rados.rados.0.smithi170.stdout:328: finishing copy_from racing read to smithi17039300-5 2023-10-20T03:36:15.335 INFO:tasks.rados.rados.0.smithi170.stdout:324: finishing write tid 1 to smithi17039300-20 2023-10-20T03:36:15.336 INFO:tasks.rados.rados.0.smithi170.stdout:324: finishing write tid 2 to smithi17039300-20 2023-10-20T03:36:15.336 INFO:tasks.rados.rados.0.smithi170.stdout:324: finishing write tid 3 to smithi17039300-20 2023-10-20T03:36:15.336 INFO:tasks.rados.rados.0.smithi170.stdout:324: finishing write tid 4 to smithi17039300-20 2023-10-20T03:36:15.336 INFO:tasks.rados.rados.0.smithi170.stdout:324: finishing write tid 5 to smithi17039300-20 2023-10-20T03:36:15.336 INFO:tasks.rados.rados.0.smithi170.stdout:324: finishing write tid 6 to smithi17039300-20 2023-10-20T03:36:15.337 INFO:tasks.rados.rados.0.smithi170.stdout:324: oid 20 updating version 0 to 48 2023-10-20T03:36:15.337 INFO:tasks.rados.rados.0.smithi170.stdout:324: oid 20 version 48 is already newer than 47 2023-10-20T03:36:15.337 INFO:tasks.rados.rados.0.smithi170.stdout:324: oid 20 updating version 48 to 50 2023-10-20T03:36:15.337 INFO:tasks.rados.rados.0.smithi170.stdout:324: oid 20 version 50 is already newer than 49 2023-10-20T03:36:15.337 INFO:tasks.rados.rados.0.smithi170.stdout:324: oid 20 updating version 50 to 51 2023-10-20T03:36:15.337 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 20 v 51 (ObjNum 136 snap 30 seq_num 136) dirty exists 2023-10-20T03:36:15.338 INFO:tasks.rados.rados.0.smithi170.stdout:324: left oid 20 (ObjNum 136 snap 30 seq_num 136) 2023-10-20T03:36:15.338 INFO:tasks.rados.rados.0.smithi170.stdout:332: expect (ObjNum 2 snap 0 seq_num 2) 2023-10-20T03:36:15.338 INFO:tasks.rados.rados.0.smithi170.stdout:328: finishing copy_from to smithi17039300-5 2023-10-20T03:36:15.338 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 62 (ObjNum 123 snap 25 seq_num 123) dirty exists 2023-10-20T03:36:15.340 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:15.341 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:15.354 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-10-20T03:36:15.434 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15] out_osds: [12] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 10, 14, 3, 7, 11, 15, 6] 2023-10-20T03:36:15.434 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-10-20T03:36:15.435 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-10-20T03:36:15.435 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-10-20T03:36:15.518 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:15.519 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:15.523 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:15.517+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:15.532 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-10-20T03:36:15.624 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:36:15.620+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:36:15.694 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:15.694 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:15.708 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-10-20T03:36:15.726 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:15.724+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:15.750 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:36:15.750 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:15.750 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 93, 2023-10-20T03:36:15.750 INFO:teuthology.orchestra.run.smithi033.stdout: "fsid": "b920f165-e9e7-4c3e-8d7b-cfdc002ae589", 2023-10-20T03:36:15.751 INFO:teuthology.orchestra.run.smithi033.stdout: "created": "2023-10-20T03:34:47.883622+0000", 2023-10-20T03:36:15.751 INFO:teuthology.orchestra.run.smithi033.stdout: "modified": "2023-10-20T03:36:15.126462+0000", 2023-10-20T03:36:15.751 INFO:teuthology.orchestra.run.smithi033.stdout: "last_up_change": "2023-10-20T03:36:11.089758+0000", 2023-10-20T03:36:15.751 INFO:teuthology.orchestra.run.smithi033.stdout: "last_in_change": "2023-10-20T03:35:35.398012+0000", 2023-10-20T03:36:15.751 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-10-20T03:36:15.751 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_num": 5806080, 2023-10-20T03:36:15.752 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_set": [ 2023-10-20T03:36:15.752 INFO:teuthology.orchestra.run.smithi033.stdout: "nodeep-scrub", 2023-10-20T03:36:15.752 INFO:teuthology.orchestra.run.smithi033.stdout: "noscrub", 2023-10-20T03:36:15.752 INFO:teuthology.orchestra.run.smithi033.stdout: "pglog_hardlimit", 2023-10-20T03:36:15.752 INFO:teuthology.orchestra.run.smithi033.stdout: "purged_snapdirs", 2023-10-20T03:36:15.753 INFO:teuthology.orchestra.run.smithi033.stdout: "recovery_deletes", 2023-10-20T03:36:15.753 INFO:teuthology.orchestra.run.smithi033.stdout: "sortbitwise" 2023-10-20T03:36:15.753 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-10-20T03:36:15.753 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_version": 7, 2023-10-20T03:36:15.753 INFO:teuthology.orchestra.run.smithi033.stdout: "full_ratio": 0.94999998807907104, 2023-10-20T03:36:15.753 INFO:teuthology.orchestra.run.smithi033.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-10-20T03:36:15.754 INFO:teuthology.orchestra.run.smithi033.stdout: "nearfull_ratio": 0.85000002384185791, 2023-10-20T03:36:15.754 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_snapshot": "", 2023-10-20T03:36:15.754 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_max": 3, 2023-10-20T03:36:15.754 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd": 16, 2023-10-20T03:36:15.754 INFO:teuthology.orchestra.run.smithi033.stdout: "require_min_compat_client": "luminous", 2023-10-20T03:36:15.755 INFO:teuthology.orchestra.run.smithi033.stdout: "min_compat_client": "luminous", 2023-10-20T03:36:15.755 INFO:teuthology.orchestra.run.smithi033.stdout: "require_osd_release": "reef", 2023-10-20T03:36:15.755 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_crimson": false, 2023-10-20T03:36:15.755 INFO:teuthology.orchestra.run.smithi033.stdout: "pools": [ 2023-10-20T03:36:15.755 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.755 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 1, 2023-10-20T03:36:15.756 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": ".mgr", 2023-10-20T03:36:15.756 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2023-10-20T03:35:00.991869+0000", 2023-10-20T03:36:15.756 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2023-10-20T03:36:15.756 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2023-10-20T03:36:15.756 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2023-10-20T03:36:15.756 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 2, 2023-10-20T03:36:15.757 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 1, 2023-10-20T03:36:15.757 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2023-10-20T03:36:15.757 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2023-10-20T03:36:15.757 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2023-10-20T03:36:15.757 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2023-10-20T03:36:15.757 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-10-20T03:36:15.758 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2023-10-20T03:36:15.758 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2023-10-20T03:36:15.758 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 1, 2023-10-20T03:36:15.758 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 1, 2023-10-20T03:36:15.758 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 1, 2023-10-20T03:36:15.759 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 1, 2023-10-20T03:36:15.759 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 1, 2023-10-20T03:36:15.759 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2023-10-20T03:36:15.759 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2023-10-20T03:36:15.759 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2023-10-20T03:36:15.759 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2023-10-20T03:36:15.760 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2023-10-20T03:36:15.760 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2023-10-20T03:36:15.760 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2023-10-20T03:36:15.760 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.760 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "25", 2023-10-20T03:36:15.760 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2023-10-20T03:36:15.761 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2023-10-20T03:36:15.761 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2023-10-20T03:36:15.761 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2023-10-20T03:36:15.761 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2023-10-20T03:36:15.761 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2023-10-20T03:36:15.762 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2023-10-20T03:36:15.762 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2023-10-20T03:36:15.762 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2023-10-20T03:36:15.762 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2023-10-20T03:36:15.762 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2023-10-20T03:36:15.762 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2023-10-20T03:36:15.763 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2023-10-20T03:36:15.763 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2023-10-20T03:36:15.763 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2023-10-20T03:36:15.763 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2023-10-20T03:36:15.763 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2023-10-20T03:36:15.764 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2023-10-20T03:36:15.764 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-10-20T03:36:15.764 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-10-20T03:36:15.764 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2023-10-20T03:36:15.764 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2023-10-20T03:36:15.764 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2023-10-20T03:36:15.765 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2023-10-20T03:36:15.765 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2023-10-20T03:36:15.765 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2023-10-20T03:36:15.765 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.765 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2023-10-20T03:36:15.765 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2023-10-20T03:36:15.766 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2023-10-20T03:36:15.766 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2023-10-20T03:36:15.766 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2023-10-20T03:36:15.766 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2023-10-20T03:36:15.766 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2023-10-20T03:36:15.766 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2023-10-20T03:36:15.767 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2023-10-20T03:36:15.767 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2023-10-20T03:36:15.767 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2023-10-20T03:36:15.767 INFO:teuthology.orchestra.run.smithi033.stdout: "options": { 2023-10-20T03:36:15.767 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_max": 32, 2023-10-20T03:36:15.768 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_min": 1 2023-10-20T03:36:15.768 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.768 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2023-10-20T03:36:15.768 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr": {} 2023-10-20T03:36:15.768 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.768 INFO:teuthology.orchestra.run.smithi033.stdout: "read_balance": { 2023-10-20T03:36:15.769 INFO:teuthology.orchestra.run.smithi033.stdout: "score_acting": 15.380000114440918, 2023-10-20T03:36:15.769 INFO:teuthology.orchestra.run.smithi033.stdout: "score_stable": 15.380000114440918, 2023-10-20T03:36:15.769 INFO:teuthology.orchestra.run.smithi033.stdout: "optimal_score": 0.12999999523162842, 2023-10-20T03:36:15.769 INFO:teuthology.orchestra.run.smithi033.stdout: "raw_score_acting": 2, 2023-10-20T03:36:15.769 INFO:teuthology.orchestra.run.smithi033.stdout: "raw_score_stable": 2, 2023-10-20T03:36:15.769 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity_weighted": 0.93946552276611328, 2023-10-20T03:36:15.770 INFO:teuthology.orchestra.run.smithi033.stdout: "average_primary_affinity": 0.93999999761581421, 2023-10-20T03:36:15.770 INFO:teuthology.orchestra.run.smithi033.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2023-10-20T03:36:15.770 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.770 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.770 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.771 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 2, 2023-10-20T03:36:15.771 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "rbd", 2023-10-20T03:36:15.771 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2023-10-20T03:35:03.136216+0000", 2023-10-20T03:36:15.771 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2023-10-20T03:36:15.771 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool", 2023-10-20T03:36:15.771 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 1, 2023-10-20T03:36:15.772 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 2, 2023-10-20T03:36:15.772 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 1, 2023-10-20T03:36:15.772 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 0, 2023-10-20T03:36:15.772 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2023-10-20T03:36:15.772 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2023-10-20T03:36:15.772 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2023-10-20T03:36:15.773 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-10-20T03:36:15.773 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2023-10-20T03:36:15.773 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2023-10-20T03:36:15.773 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 8, 2023-10-20T03:36:15.773 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 8, 2023-10-20T03:36:15.773 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 8, 2023-10-20T03:36:15.774 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 8, 2023-10-20T03:36:15.774 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 8, 2023-10-20T03:36:15.774 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2023-10-20T03:36:15.774 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2023-10-20T03:36:15.774 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2023-10-20T03:36:15.774 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2023-10-20T03:36:15.775 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2023-10-20T03:36:15.775 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2023-10-20T03:36:15.775 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2023-10-20T03:36:15.775 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.775 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "28", 2023-10-20T03:36:15.775 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2023-10-20T03:36:15.776 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2023-10-20T03:36:15.776 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2023-10-20T03:36:15.776 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2023-10-20T03:36:15.776 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2023-10-20T03:36:15.776 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 0, 2023-10-20T03:36:15.776 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 0, 2023-10-20T03:36:15.777 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2023-10-20T03:36:15.777 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2023-10-20T03:36:15.777 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2023-10-20T03:36:15.777 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2023-10-20T03:36:15.777 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2023-10-20T03:36:15.777 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2023-10-20T03:36:15.778 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2023-10-20T03:36:15.778 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2023-10-20T03:36:15.778 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2023-10-20T03:36:15.778 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2023-10-20T03:36:15.778 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2023-10-20T03:36:15.779 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-10-20T03:36:15.779 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-10-20T03:36:15.779 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2023-10-20T03:36:15.779 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2023-10-20T03:36:15.779 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2023-10-20T03:36:15.779 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "", 2023-10-20T03:36:15.780 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2023-10-20T03:36:15.780 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2023-10-20T03:36:15.780 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.780 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2023-10-20T03:36:15.780 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2023-10-20T03:36:15.780 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2023-10-20T03:36:15.781 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2023-10-20T03:36:15.781 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2023-10-20T03:36:15.781 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2023-10-20T03:36:15.781 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2023-10-20T03:36:15.781 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2023-10-20T03:36:15.781 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 0, 2023-10-20T03:36:15.782 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2023-10-20T03:36:15.782 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2023-10-20T03:36:15.782 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2023-10-20T03:36:15.782 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2023-10-20T03:36:15.782 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd": {} 2023-10-20T03:36:15.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.783 INFO:teuthology.orchestra.run.smithi033.stdout: "read_balance": { 2023-10-20T03:36:15.783 INFO:teuthology.orchestra.run.smithi033.stdout: "score_acting": 5.9499998092651367, 2023-10-20T03:36:15.783 INFO:teuthology.orchestra.run.smithi033.stdout: "score_stable": 5.9499998092651367, 2023-10-20T03:36:15.783 INFO:teuthology.orchestra.run.smithi033.stdout: "optimal_score": 0.62999999523162842, 2023-10-20T03:36:15.783 INFO:teuthology.orchestra.run.smithi033.stdout: "raw_score_acting": 3.75, 2023-10-20T03:36:15.784 INFO:teuthology.orchestra.run.smithi033.stdout: "raw_score_stable": 3.75, 2023-10-20T03:36:15.784 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity_weighted": 0.93946552276611328, 2023-10-20T03:36:15.784 INFO:teuthology.orchestra.run.smithi033.stdout: "average_primary_affinity": 0.93999999761581421, 2023-10-20T03:36:15.784 INFO:teuthology.orchestra.run.smithi033.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2023-10-20T03:36:15.784 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.784 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.785 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 3, 2023-10-20T03:36:15.785 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": "unique_pool_0", 2023-10-20T03:36:15.785 INFO:teuthology.orchestra.run.smithi033.stdout: "create_time": "2023-10-20T03:35:12.214840+0000", 2023-10-20T03:36:15.785 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 8193, 2023-10-20T03:36:15.785 INFO:teuthology.orchestra.run.smithi033.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2023-10-20T03:36:15.786 INFO:teuthology.orchestra.run.smithi033.stdout: "type": 3, 2023-10-20T03:36:15.786 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 7, 2023-10-20T03:36:15.786 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": 5, 2023-10-20T03:36:15.786 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_rule": 1, 2023-10-20T03:36:15.786 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_count": 0, 2023-10-20T03:36:15.786 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_target": 0, 2023-10-20T03:36:15.787 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_barrier": 0, 2023-10-20T03:36:15.787 INFO:teuthology.orchestra.run.smithi033.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-10-20T03:36:15.787 INFO:teuthology.orchestra.run.smithi033.stdout: "object_hash": 2, 2023-10-20T03:36:15.787 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscale_mode": "off", 2023-10-20T03:36:15.787 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": 16, 2023-10-20T03:36:15.787 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num": 16, 2023-10-20T03:36:15.788 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_placement_num_target": 16, 2023-10-20T03:36:15.788 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_target": 16, 2023-10-20T03:36:15.788 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num_pending": 16, 2023-10-20T03:36:15.788 INFO:teuthology.orchestra.run.smithi033.stdout: "last_pg_merge_meta": { 2023-10-20T03:36:15.788 INFO:teuthology.orchestra.run.smithi033.stdout: "source_pgid": "0.0", 2023-10-20T03:36:15.789 INFO:teuthology.orchestra.run.smithi033.stdout: "ready_epoch": 0, 2023-10-20T03:36:15.789 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_started": 0, 2023-10-20T03:36:15.789 INFO:teuthology.orchestra.run.smithi033.stdout: "last_epoch_clean": 0, 2023-10-20T03:36:15.789 INFO:teuthology.orchestra.run.smithi033.stdout: "source_version": "0'0", 2023-10-20T03:36:15.789 INFO:teuthology.orchestra.run.smithi033.stdout: "target_version": "0'0" 2023-10-20T03:36:15.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.790 INFO:teuthology.orchestra.run.smithi033.stdout: "last_change": "92", 2023-10-20T03:36:15.790 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend": "0", 2023-10-20T03:36:15.790 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_prenautilus": "0", 2023-10-20T03:36:15.790 INFO:teuthology.orchestra.run.smithi033.stdout: "last_force_op_resend_preluminous": "0", 2023-10-20T03:36:15.790 INFO:teuthology.orchestra.run.smithi033.stdout: "auid": 0, 2023-10-20T03:36:15.790 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_mode": "selfmanaged", 2023-10-20T03:36:15.791 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_seq": 58, 2023-10-20T03:36:15.791 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_epoch": 92, 2023-10-20T03:36:15.791 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_snaps": [], 2023-10-20T03:36:15.791 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps": "[]", 2023-10-20T03:36:15.791 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_bytes": 0, 2023-10-20T03:36:15.791 INFO:teuthology.orchestra.run.smithi033.stdout: "quota_max_objects": 0, 2023-10-20T03:36:15.792 INFO:teuthology.orchestra.run.smithi033.stdout: "tiers": [], 2023-10-20T03:36:15.792 INFO:teuthology.orchestra.run.smithi033.stdout: "tier_of": -1, 2023-10-20T03:36:15.792 INFO:teuthology.orchestra.run.smithi033.stdout: "read_tier": -1, 2023-10-20T03:36:15.792 INFO:teuthology.orchestra.run.smithi033.stdout: "write_tier": -1, 2023-10-20T03:36:15.792 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_mode": "none", 2023-10-20T03:36:15.792 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_bytes": 0, 2023-10-20T03:36:15.793 INFO:teuthology.orchestra.run.smithi033.stdout: "target_max_objects": 0, 2023-10-20T03:36:15.793 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-10-20T03:36:15.793 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-10-20T03:36:15.793 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_target_full_ratio_micro": 800000, 2023-10-20T03:36:15.793 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_flush_age": 0, 2023-10-20T03:36:15.794 INFO:teuthology.orchestra.run.smithi033.stdout: "cache_min_evict_age": 0, 2023-10-20T03:36:15.794 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profile": "shecprofile", 2023-10-20T03:36:15.794 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_params": { 2023-10-20T03:36:15.794 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "none" 2023-10-20T03:36:15.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.794 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_period": 0, 2023-10-20T03:36:15.795 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_count": 0, 2023-10-20T03:36:15.795 INFO:teuthology.orchestra.run.smithi033.stdout: "use_gmt_hitset": true, 2023-10-20T03:36:15.795 INFO:teuthology.orchestra.run.smithi033.stdout: "min_read_recency_for_promote": 0, 2023-10-20T03:36:15.795 INFO:teuthology.orchestra.run.smithi033.stdout: "min_write_recency_for_promote": 0, 2023-10-20T03:36:15.795 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_grade_decay_rate": 0, 2023-10-20T03:36:15.795 INFO:teuthology.orchestra.run.smithi033.stdout: "hit_set_search_last_n": 0, 2023-10-20T03:36:15.796 INFO:teuthology.orchestra.run.smithi033.stdout: "grade_table": [], 2023-10-20T03:36:15.796 INFO:teuthology.orchestra.run.smithi033.stdout: "stripe_width": 16384, 2023-10-20T03:36:15.796 INFO:teuthology.orchestra.run.smithi033.stdout: "expected_num_objects": 0, 2023-10-20T03:36:15.796 INFO:teuthology.orchestra.run.smithi033.stdout: "fast_read": false, 2023-10-20T03:36:15.796 INFO:teuthology.orchestra.run.smithi033.stdout: "options": {}, 2023-10-20T03:36:15.796 INFO:teuthology.orchestra.run.smithi033.stdout: "application_metadata": { 2023-10-20T03:36:15.797 INFO:teuthology.orchestra.run.smithi033.stdout: "rados": {} 2023-10-20T03:36:15.797 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.797 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.797 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-10-20T03:36:15.797 INFO:teuthology.orchestra.run.smithi033.stdout: "osds": [ 2023-10-20T03:36:15.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.798 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2023-10-20T03:36:15.798 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "4872451c-1bc7-4f21-a95f-986fa428ea65", 2023-10-20T03:36:15.798 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.798 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.798 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.799 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0.072998046875, 2023-10-20T03:36:15.799 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.799 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.799 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.799 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 55, 2023-10-20T03:36:15.799 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.800 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.800 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.800 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.800 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.800 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.801 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6824", 2023-10-20T03:36:15.801 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:36:15.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.801 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.801 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6825", 2023-10-20T03:36:15.802 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:36:15.802 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.802 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.802 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.802 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.803 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.803 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6826", 2023-10-20T03:36:15.803 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:36:15.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.804 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.804 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.804 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6827", 2023-10-20T03:36:15.804 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:36:15.804 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.804 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.805 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.805 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.805 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.805 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6830", 2023-10-20T03:36:15.806 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:36:15.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.806 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.806 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6831", 2023-10-20T03:36:15.807 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:36:15.807 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.807 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.807 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.807 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.808 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.808 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6828", 2023-10-20T03:36:15.808 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:36:15.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.809 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.809 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6829", 2023-10-20T03:36:15.809 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 121544157 2023-10-20T03:36:15.809 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.809 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.810 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6825/121544157", 2023-10-20T03:36:15.810 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6827/121544157", 2023-10-20T03:36:15.810 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6831/121544157", 2023-10-20T03:36:15.810 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6829/121544157", 2023-10-20T03:36:15.810 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.810 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.811 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.811 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.811 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2023-10-20T03:36:15.812 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "0b570426-e165-4caa-a914-e1bed7d805fc", 2023-10-20T03:36:15.812 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.812 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.812 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.812 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.812 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.813 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.813 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.813 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 92, 2023-10-20T03:36:15.813 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.813 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.813 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.814 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.814 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.814 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6808", 2023-10-20T03:36:15.814 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:36:15.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.815 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.815 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.815 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6809", 2023-10-20T03:36:15.815 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:36:15.816 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.816 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.816 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.816 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.817 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.817 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6810", 2023-10-20T03:36:15.817 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:36:15.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.817 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.818 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6811", 2023-10-20T03:36:15.818 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:36:15.818 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.818 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.818 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.819 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.819 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.819 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.819 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6818", 2023-10-20T03:36:15.819 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:36:15.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.820 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.820 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.820 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6820", 2023-10-20T03:36:15.820 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:36:15.820 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.820 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.821 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.821 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.821 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.821 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6813", 2023-10-20T03:36:15.822 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:36:15.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.822 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.822 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.822 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6815", 2023-10-20T03:36:15.822 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 562043184 2023-10-20T03:36:15.823 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.823 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.823 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.40:6809/562043184", 2023-10-20T03:36:15.823 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.40:6811/562043184", 2023-10-20T03:36:15.823 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.40:6820/562043184", 2023-10-20T03:36:15.824 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.40:6815/562043184", 2023-10-20T03:36:15.824 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.824 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.824 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.824 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.825 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2023-10-20T03:36:15.825 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "e0184c10-844b-4bcb-adaf-a0d407b708ee", 2023-10-20T03:36:15.825 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.825 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.825 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.826 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.826 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.826 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.826 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.826 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2023-10-20T03:36:15.826 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.827 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.827 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.827 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.827 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.827 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.828 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6824", 2023-10-20T03:36:15.828 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:36:15.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.828 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.828 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.828 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6825", 2023-10-20T03:36:15.829 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:36:15.829 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.829 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.829 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.829 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.830 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.830 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6826", 2023-10-20T03:36:15.830 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:36:15.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.831 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.831 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6827", 2023-10-20T03:36:15.831 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:36:15.831 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.831 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.832 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.832 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.832 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.832 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.832 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.833 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6830", 2023-10-20T03:36:15.833 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:36:15.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.833 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.833 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6831", 2023-10-20T03:36:15.834 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:36:15.834 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.834 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.834 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.835 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.835 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.835 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6828", 2023-10-20T03:36:15.835 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:36:15.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.836 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.836 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6829", 2023-10-20T03:36:15.836 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3785405088 2023-10-20T03:36:15.837 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.837 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.837 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.64:6825/3785405088", 2023-10-20T03:36:15.837 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.64:6827/3785405088", 2023-10-20T03:36:15.837 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.64:6831/3785405088", 2023-10-20T03:36:15.838 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.64:6829/3785405088", 2023-10-20T03:36:15.838 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.838 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.838 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.838 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.839 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2023-10-20T03:36:15.839 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "8120023b-1d48-411d-a642-2529a697b5c9", 2023-10-20T03:36:15.839 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.839 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.840 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.840 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 0.9584503173828125, 2023-10-20T03:36:15.840 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.840 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.840 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.840 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 92, 2023-10-20T03:36:15.841 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.841 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.841 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.841 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.841 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.841 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.842 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6818", 2023-10-20T03:36:15.842 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:36:15.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.842 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.843 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6820", 2023-10-20T03:36:15.843 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:36:15.843 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.843 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.843 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.844 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.844 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.844 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6821", 2023-10-20T03:36:15.844 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:36:15.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.845 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.845 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6823", 2023-10-20T03:36:15.845 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:36:15.845 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.846 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.846 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.846 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.847 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.847 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6829", 2023-10-20T03:36:15.847 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:36:15.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.847 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.848 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6831", 2023-10-20T03:36:15.848 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:36:15.848 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.848 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.848 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.849 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.849 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.849 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6825", 2023-10-20T03:36:15.849 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:36:15.850 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.850 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.850 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.850 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6827", 2023-10-20T03:36:15.850 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1426674535 2023-10-20T03:36:15.850 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.851 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.851 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.170:6820/1426674535", 2023-10-20T03:36:15.851 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.170:6823/1426674535", 2023-10-20T03:36:15.851 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.170:6831/1426674535", 2023-10-20T03:36:15.851 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.170:6827/1426674535", 2023-10-20T03:36:15.852 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.852 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.852 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.852 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.853 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2023-10-20T03:36:15.853 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "549b3b39-41d5-4d9b-932e-a7b3e8cc0684", 2023-10-20T03:36:15.853 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.853 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.853 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.854 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.854 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.854 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.854 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.854 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 55, 2023-10-20T03:36:15.854 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.855 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.855 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.855 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.855 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.855 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6808", 2023-10-20T03:36:15.856 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:36:15.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.856 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.856 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6809", 2023-10-20T03:36:15.856 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:36:15.857 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.857 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.857 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.857 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.858 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.858 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6810", 2023-10-20T03:36:15.858 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:36:15.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.858 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.859 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6811", 2023-10-20T03:36:15.859 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:36:15.859 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.859 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.859 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.860 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.860 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.860 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6814", 2023-10-20T03:36:15.860 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:36:15.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.861 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.861 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6815", 2023-10-20T03:36:15.861 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:36:15.861 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.861 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.862 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.862 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.862 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.862 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.862 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6812", 2023-10-20T03:36:15.863 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:36:15.863 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.863 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.863 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6813", 2023-10-20T03:36:15.863 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3186493546 2023-10-20T03:36:15.864 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.864 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.864 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6809/3186493546", 2023-10-20T03:36:15.864 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6811/3186493546", 2023-10-20T03:36:15.864 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6815/3186493546", 2023-10-20T03:36:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6813/3186493546", 2023-10-20T03:36:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2023-10-20T03:36:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "a99ea3db-6433-4312-83d1-64ba69f028a4", 2023-10-20T03:36:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.866 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.867 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.867 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.867 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.867 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 21, 2023-10-20T03:36:15.867 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 0, 2023-10-20T03:36:15.867 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.867 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.868 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.868 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.868 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.868 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6814", 2023-10-20T03:36:15.868 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:36:15.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.869 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.869 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6817", 2023-10-20T03:36:15.869 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:36:15.869 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.870 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.870 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.870 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.871 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.871 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6821", 2023-10-20T03:36:15.871 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:36:15.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.871 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.872 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6823", 2023-10-20T03:36:15.872 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:36:15.872 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.872 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.872 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.873 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.873 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.873 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6829", 2023-10-20T03:36:15.873 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:36:15.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.874 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.874 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6831", 2023-10-20T03:36:15.874 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:36:15.874 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.874 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.875 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.875 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.875 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.875 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6825", 2023-10-20T03:36:15.876 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:36:15.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.876 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.876 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6827", 2023-10-20T03:36:15.876 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1211670976 2023-10-20T03:36:15.877 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.877 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.877 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.40:6817/1211670976", 2023-10-20T03:36:15.877 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.40:6823/1211670976", 2023-10-20T03:36:15.878 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.40:6831/1211670976", 2023-10-20T03:36:15.878 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.40:6827/1211670976", 2023-10-20T03:36:15.878 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.878 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.878 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.878 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.879 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2023-10-20T03:36:15.879 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "d2b38063-49eb-4f56-bb8d-411419b71b9b", 2023-10-20T03:36:15.879 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.879 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.880 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.880 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.880 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 22, 2023-10-20T03:36:15.880 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 28, 2023-10-20T03:36:15.880 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 89, 2023-10-20T03:36:15.880 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 92, 2023-10-20T03:36:15.881 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 29, 2023-10-20T03:36:15.881 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.881 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.881 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.881 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.882 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6808", 2023-10-20T03:36:15.882 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2043848066 2023-10-20T03:36:15.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.882 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.882 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6809", 2023-10-20T03:36:15.883 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2043848066 2023-10-20T03:36:15.883 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.883 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.883 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.883 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.884 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.884 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6810", 2023-10-20T03:36:15.884 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2043848066 2023-10-20T03:36:15.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.885 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.885 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6811", 2023-10-20T03:36:15.885 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2043848066 2023-10-20T03:36:15.885 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.885 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.886 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.886 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.886 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.886 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6814", 2023-10-20T03:36:15.887 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2043848066 2023-10-20T03:36:15.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.887 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.887 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6815", 2023-10-20T03:36:15.888 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2043848066 2023-10-20T03:36:15.888 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.888 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.888 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.888 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.889 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.889 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6812", 2023-10-20T03:36:15.889 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2043848066 2023-10-20T03:36:15.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.890 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.890 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6813", 2023-10-20T03:36:15.890 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2043848066 2023-10-20T03:36:15.890 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.890 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.891 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.64:6809/2043848066", 2023-10-20T03:36:15.891 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.64:6811/2043848066", 2023-10-20T03:36:15.891 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.64:6815/2043848066", 2023-10-20T03:36:15.891 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.64:6813/2043848066", 2023-10-20T03:36:15.891 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.891 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.892 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.892 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.892 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2023-10-20T03:36:15.892 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "80e34f64-35dc-4a6d-9f7f-c2263c78350d", 2023-10-20T03:36:15.893 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.893 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.893 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.893 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.893 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.893 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.894 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.894 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 31, 2023-10-20T03:36:15.894 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.894 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.894 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.894 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.895 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.895 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6810", 2023-10-20T03:36:15.895 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:36:15.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.896 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.896 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6811", 2023-10-20T03:36:15.896 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:36:15.896 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.896 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.897 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.897 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.897 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.897 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6812", 2023-10-20T03:36:15.897 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:36:15.898 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.898 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.898 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.898 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6813", 2023-10-20T03:36:15.898 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:36:15.899 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.899 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.899 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.899 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.899 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.900 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6816", 2023-10-20T03:36:15.900 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:36:15.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.900 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.900 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6817", 2023-10-20T03:36:15.901 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:36:15.901 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.901 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.901 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.902 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.902 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.902 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.902 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6814", 2023-10-20T03:36:15.902 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:36:15.902 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.903 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.903 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.903 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6815", 2023-10-20T03:36:15.903 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2804292733 2023-10-20T03:36:15.903 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.903 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.904 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.904 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.170:6811/2804292733", 2023-10-20T03:36:15.904 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.170:6813/2804292733", 2023-10-20T03:36:15.904 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.170:6817/2804292733", 2023-10-20T03:36:15.904 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.170:6815/2804292733", 2023-10-20T03:36:15.904 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.905 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.905 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.905 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.905 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.905 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 8, 2023-10-20T03:36:15.906 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "1cd10625-66a8-4488-b1e5-378cb0062107", 2023-10-20T03:36:15.906 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.906 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.906 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.906 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.907 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.907 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.907 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.907 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 31, 2023-10-20T03:36:15.907 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.907 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.908 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.908 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.908 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.908 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.908 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6800", 2023-10-20T03:36:15.908 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:36:15.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.909 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.909 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.909 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6801", 2023-10-20T03:36:15.909 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:36:15.909 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.909 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.910 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.910 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.910 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.910 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.910 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.911 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6802", 2023-10-20T03:36:15.911 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:36:15.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.911 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.911 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6803", 2023-10-20T03:36:15.912 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:36:15.912 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.912 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.912 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.912 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.913 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.913 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.913 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6806", 2023-10-20T03:36:15.913 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:36:15.913 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.913 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.914 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.914 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6807", 2023-10-20T03:36:15.914 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:36:15.914 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.914 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.914 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.915 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.915 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.915 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.915 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.915 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6804", 2023-10-20T03:36:15.915 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:36:15.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.916 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.916 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.916 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6805", 2023-10-20T03:36:15.916 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1638010589 2023-10-20T03:36:15.916 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.917 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.917 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6801/1638010589", 2023-10-20T03:36:15.917 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6803/1638010589", 2023-10-20T03:36:15.917 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6807/1638010589", 2023-10-20T03:36:15.917 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6805/1638010589", 2023-10-20T03:36:15.918 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.918 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.918 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.918 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.919 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 9, 2023-10-20T03:36:15.919 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4", 2023-10-20T03:36:15.919 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.919 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.919 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.919 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.920 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.920 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.920 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.920 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 92, 2023-10-20T03:36:15.920 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.920 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.921 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.921 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.921 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.921 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.921 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6800", 2023-10-20T03:36:15.921 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:36:15.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.922 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.922 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.922 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6801", 2023-10-20T03:36:15.922 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:36:15.923 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.923 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.923 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.923 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.923 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.924 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.924 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6802", 2023-10-20T03:36:15.924 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:36:15.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.925 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.925 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6803", 2023-10-20T03:36:15.925 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:36:15.925 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.925 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.925 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.926 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.926 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.926 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.926 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6806", 2023-10-20T03:36:15.926 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:36:15.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.927 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.927 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6807", 2023-10-20T03:36:15.927 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:36:15.927 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.928 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.928 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.928 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.928 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.929 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6804", 2023-10-20T03:36:15.929 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:36:15.929 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.929 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.929 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.929 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6805", 2023-10-20T03:36:15.930 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2173797306 2023-10-20T03:36:15.930 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.930 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.930 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.930 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.40:6801/2173797306", 2023-10-20T03:36:15.930 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.40:6803/2173797306", 2023-10-20T03:36:15.931 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.40:6807/2173797306", 2023-10-20T03:36:15.931 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.40:6805/2173797306", 2023-10-20T03:36:15.931 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.931 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.931 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.931 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.932 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.932 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 10, 2023-10-20T03:36:15.932 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "42a67bcc-5c07-499d-8902-08c2f15e7fbb", 2023-10-20T03:36:15.932 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.932 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.933 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.933 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.933 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.933 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.933 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.933 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 56, 2023-10-20T03:36:15.934 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.934 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.934 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.934 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.934 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.935 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.935 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6800", 2023-10-20T03:36:15.935 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:36:15.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.935 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.935 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.935 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6801", 2023-10-20T03:36:15.936 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:36:15.936 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.936 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.936 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.936 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.937 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.937 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.937 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6802", 2023-10-20T03:36:15.937 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:36:15.937 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.938 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.938 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.938 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6803", 2023-10-20T03:36:15.938 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:36:15.938 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.938 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.939 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.939 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.939 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.939 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6806", 2023-10-20T03:36:15.940 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:36:15.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.940 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.940 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.940 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6807", 2023-10-20T03:36:15.940 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:36:15.941 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.941 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.941 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.941 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.941 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.942 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.942 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6804", 2023-10-20T03:36:15.942 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:36:15.942 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.942 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.942 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.943 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6805", 2023-10-20T03:36:15.943 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 91701057 2023-10-20T03:36:15.943 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.943 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.943 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.64:6801/91701057", 2023-10-20T03:36:15.944 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.64:6803/91701057", 2023-10-20T03:36:15.944 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.64:6807/91701057", 2023-10-20T03:36:15.944 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.64:6805/91701057", 2023-10-20T03:36:15.944 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.944 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.944 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.945 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.945 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.945 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 11, 2023-10-20T03:36:15.945 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "ab1db647-aada-4deb-abdc-0a1baff548db", 2023-10-20T03:36:15.945 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.946 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.946 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.946 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.946 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.946 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.946 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.947 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 92, 2023-10-20T03:36:15.947 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.947 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.947 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.947 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.947 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.948 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.948 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6802", 2023-10-20T03:36:15.948 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:36:15.948 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.948 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.948 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.949 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6803", 2023-10-20T03:36:15.949 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:36:15.949 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.949 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.949 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.949 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.950 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.950 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.950 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.950 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6804", 2023-10-20T03:36:15.950 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:36:15.951 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.951 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.951 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.951 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6805", 2023-10-20T03:36:15.951 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:36:15.951 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.952 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.952 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.952 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.952 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.952 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.952 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.953 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6808", 2023-10-20T03:36:15.953 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:36:15.953 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.953 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.953 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.953 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6809", 2023-10-20T03:36:15.954 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:36:15.954 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.954 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.954 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.954 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.954 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.955 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.955 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.955 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6806", 2023-10-20T03:36:15.955 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:36:15.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.955 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.956 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.956 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6807", 2023-10-20T03:36:15.956 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1717984751 2023-10-20T03:36:15.956 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.956 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.957 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.170:6803/1717984751", 2023-10-20T03:36:15.957 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.170:6805/1717984751", 2023-10-20T03:36:15.957 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.170:6809/1717984751", 2023-10-20T03:36:15.957 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.170:6807/1717984751", 2023-10-20T03:36:15.957 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.957 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.958 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.958 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.958 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.958 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.958 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 12, 2023-10-20T03:36:15.958 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "b20cee74-2cce-4153-97c2-56e9db40c712", 2023-10-20T03:36:15.959 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.959 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 0, 2023-10-20T03:36:15.959 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 0, 2023-10-20T03:36:15.959 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.959 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.959 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.960 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.960 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 31, 2023-10-20T03:36:15.960 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.960 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.960 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.960 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.961 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.961 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.961 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6816", 2023-10-20T03:36:15.961 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:36:15.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.961 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.961 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.962 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6817", 2023-10-20T03:36:15.962 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:36:15.962 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.962 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.963 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.963 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.963 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.963 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.963 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6818", 2023-10-20T03:36:15.963 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:36:15.964 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.964 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.964 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.964 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6819", 2023-10-20T03:36:15.964 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:36:15.964 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.965 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.965 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.965 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.965 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.965 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.965 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.966 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6822", 2023-10-20T03:36:15.966 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:36:15.966 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.966 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.966 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.966 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6823", 2023-10-20T03:36:15.967 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:36:15.967 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.967 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.967 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.967 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.967 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.968 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.968 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.968 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6820", 2023-10-20T03:36:15.968 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:36:15.968 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.968 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.969 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.969 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6821", 2023-10-20T03:36:15.969 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3797427065 2023-10-20T03:36:15.969 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.969 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.969 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.970 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.33:6817/3797427065", 2023-10-20T03:36:15.970 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.33:6819/3797427065", 2023-10-20T03:36:15.970 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.33:6823/3797427065", 2023-10-20T03:36:15.970 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.33:6821/3797427065", 2023-10-20T03:36:15.970 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.970 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.971 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.971 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.971 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.971 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.971 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 13, 2023-10-20T03:36:15.971 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "0974cd04-c690-4cba-ac05-5d404493dd1d", 2023-10-20T03:36:15.972 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.972 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.972 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.972 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.972 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.972 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.973 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.973 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 92, 2023-10-20T03:36:15.973 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.973 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.973 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.973 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.974 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.974 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.974 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6812", 2023-10-20T03:36:15.974 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:36:15.974 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.974 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.975 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.975 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6816", 2023-10-20T03:36:15.975 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:36:15.975 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.975 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.976 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.976 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.976 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.976 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.976 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6819", 2023-10-20T03:36:15.976 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:36:15.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.977 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.977 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6822", 2023-10-20T03:36:15.977 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:36:15.977 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.978 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.978 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.978 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.978 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.978 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.979 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6828", 2023-10-20T03:36:15.979 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:36:15.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.979 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.979 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6830", 2023-10-20T03:36:15.980 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:36:15.980 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.980 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.980 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.980 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.981 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.981 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.981 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6824", 2023-10-20T03:36:15.981 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:36:15.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.981 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.982 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.982 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.40:6826", 2023-10-20T03:36:15.982 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3041424233 2023-10-20T03:36:15.982 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.982 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.983 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.40:6816/3041424233", 2023-10-20T03:36:15.983 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.40:6822/3041424233", 2023-10-20T03:36:15.983 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.40:6830/3041424233", 2023-10-20T03:36:15.983 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.40:6826/3041424233", 2023-10-20T03:36:15.983 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.984 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.984 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.984 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.984 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.984 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 14, 2023-10-20T03:36:15.985 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "3bf53124-13b6-4e5c-9868-2382719ffe25", 2023-10-20T03:36:15.985 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.985 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.985 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.985 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.985 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.985 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.986 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.986 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 31, 2023-10-20T03:36:15.986 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.986 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.986 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:15.987 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.987 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.987 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.987 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6816", 2023-10-20T03:36:15.987 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:36:15.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.988 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.988 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6817", 2023-10-20T03:36:15.988 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:36:15.988 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.988 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.989 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:15.989 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.989 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.990 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6818", 2023-10-20T03:36:15.990 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:36:15.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.990 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.990 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.990 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6819", 2023-10-20T03:36:15.991 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:36:15.991 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.991 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.991 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.991 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:15.991 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.991 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.992 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.992 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6822", 2023-10-20T03:36:15.992 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:36:15.992 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.992 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.992 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.993 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6823", 2023-10-20T03:36:15.993 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:36:15.993 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.993 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.993 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.993 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:15.994 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:15.994 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.994 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:15.994 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6820", 2023-10-20T03:36:15.994 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:36:15.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.995 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.995 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:15.995 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.64:6821", 2023-10-20T03:36:15.995 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2940007286 2023-10-20T03:36:15.995 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:15.995 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.996 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.64:6817/2940007286", 2023-10-20T03:36:15.996 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.64:6819/2940007286", 2023-10-20T03:36:15.996 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.64:6823/2940007286", 2023-10-20T03:36:15.996 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.64:6821/2940007286", 2023-10-20T03:36:15.996 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:15.997 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:15.997 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:15.997 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:15.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:15.997 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:15.997 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 15, 2023-10-20T03:36:15.998 INFO:teuthology.orchestra.run.smithi033.stdout: "uuid": "b0b5ffe5-a25b-412f-818f-b60907c0089b", 2023-10-20T03:36:15.998 INFO:teuthology.orchestra.run.smithi033.stdout: "up": 1, 2023-10-20T03:36:15.998 INFO:teuthology.orchestra.run.smithi033.stdout: "in": 1, 2023-10-20T03:36:15.998 INFO:teuthology.orchestra.run.smithi033.stdout: "weight": 1, 2023-10-20T03:36:15.998 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_affinity": 1, 2023-10-20T03:36:15.998 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_begin": 0, 2023-10-20T03:36:15.999 INFO:teuthology.orchestra.run.smithi033.stdout: "last_clean_end": 0, 2023-10-20T03:36:15.999 INFO:teuthology.orchestra.run.smithi033.stdout: "up_from": 22, 2023-10-20T03:36:15.999 INFO:teuthology.orchestra.run.smithi033.stdout: "up_thru": 29, 2023-10-20T03:36:15.999 INFO:teuthology.orchestra.run.smithi033.stdout: "down_at": 0, 2023-10-20T03:36:15.999 INFO:teuthology.orchestra.run.smithi033.stdout: "lost_at": 0, 2023-10-20T03:36:15.999 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addrs": { 2023-10-20T03:36:16.000 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:16.000 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.000 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:16.000 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6819", 2023-10-20T03:36:16.000 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:36:16.000 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.001 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.001 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:16.001 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6822", 2023-10-20T03:36:16.001 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:36:16.001 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:16.001 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:16.002 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.002 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addrs": { 2023-10-20T03:36:16.002 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:16.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.002 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:16.003 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6824", 2023-10-20T03:36:16.003 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:36:16.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.003 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.005 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:16.006 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6826", 2023-10-20T03:36:16.006 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:36:16.006 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:16.006 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:16.006 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.006 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addrs": { 2023-10-20T03:36:16.007 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:16.007 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.007 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:16.007 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6832", 2023-10-20T03:36:16.007 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:36:16.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.008 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:16.008 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6833", 2023-10-20T03:36:16.008 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:36:16.008 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:16.008 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:16.009 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.009 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addrs": { 2023-10-20T03:36:16.009 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2023-10-20T03:36:16.009 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.009 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2023-10-20T03:36:16.009 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6828", 2023-10-20T03:36:16.010 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:36:16.010 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.010 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.010 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2023-10-20T03:36:16.010 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.170:6830", 2023-10-20T03:36:16.010 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 826332016 2023-10-20T03:36:16.011 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:16.011 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:16.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.011 INFO:teuthology.orchestra.run.smithi033.stdout: "public_addr": "172.21.15.170:6822/826332016", 2023-10-20T03:36:16.011 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_addr": "172.21.15.170:6826/826332016", 2023-10-20T03:36:16.011 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_back_addr": "172.21.15.170:6833/826332016", 2023-10-20T03:36:16.012 INFO:teuthology.orchestra.run.smithi033.stdout: "heartbeat_front_addr": "172.21.15.170:6830/826332016", 2023-10-20T03:36:16.012 INFO:teuthology.orchestra.run.smithi033.stdout: "state": [ 2023-10-20T03:36:16.012 INFO:teuthology.orchestra.run.smithi033.stdout: "exists", 2023-10-20T03:36:16.012 INFO:teuthology.orchestra.run.smithi033.stdout: "up" 2023-10-20T03:36:16.012 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:16.012 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:16.013 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-10-20T03:36:16.013 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_xinfo": [ 2023-10-20T03:36:16.013 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.013 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 0, 2023-10-20T03:36:16.013 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.013 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.014 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.014 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.014 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.014 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.204996+0000", 2023-10-20T03:36:16.014 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.015 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.015 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 1, 2023-10-20T03:36:16.015 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.015 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.015 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.015 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.016 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.016 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:36:11.884878+0000", 2023-10-20T03:36:16.016 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.016 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.016 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 2, 2023-10-20T03:36:16.017 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.017 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.017 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.017 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.017 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.017 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:36:11.632626+0000", 2023-10-20T03:36:16.018 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.018 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.018 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 3, 2023-10-20T03:36:16.018 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.018 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.019 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.019 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.019 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.019 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.175698+0000", 2023-10-20T03:36:16.019 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.020 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.020 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 4, 2023-10-20T03:36:16.020 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.020 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.020 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.020 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.021 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.021 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:36:03.397982+0000", 2023-10-20T03:36:16.021 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.021 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.021 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 5, 2023-10-20T03:36:16.022 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.022 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.022 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.022 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.022 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.023 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:36:03.463084+0000", 2023-10-20T03:36:16.023 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.023 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.023 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 6, 2023-10-20T03:36:16.023 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.023 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.024 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.024 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.024 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.024 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:36:10.320469+0000", 2023-10-20T03:36:16.024 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 29 2023-10-20T03:36:16.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.025 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.025 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 7, 2023-10-20T03:36:16.025 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.025 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.025 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.025 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.026 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.026 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:35:58.321221+0000", 2023-10-20T03:36:16.026 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.026 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.026 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 8, 2023-10-20T03:36:16.027 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.027 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.027 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.027 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.027 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.027 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:36:10.321913+0000", 2023-10-20T03:36:16.028 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.028 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.028 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 9, 2023-10-20T03:36:16.028 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.028 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.029 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.029 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.029 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.029 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:35:59.745876+0000", 2023-10-20T03:36:16.029 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.029 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.030 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.030 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 10, 2023-10-20T03:36:16.030 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.030 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.030 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.031 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.031 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.031 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:36:02.082859+0000", 2023-10-20T03:36:16.031 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.031 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.031 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.031 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 11, 2023-10-20T03:36:16.032 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.032 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.032 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.032 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.032 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.033 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:36:03.991453+0000", 2023-10-20T03:36:16.033 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.033 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.033 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.033 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 12, 2023-10-20T03:36:16.033 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.034 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.034 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.034 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.034 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 65536, 2023-10-20T03:36:16.034 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:36:05.214755+0000", 2023-10-20T03:36:16.034 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.035 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 13, 2023-10-20T03:36:16.035 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.035 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.035 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.036 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.036 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.036 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:36:01.467282+0000", 2023-10-20T03:36:16.036 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.036 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.037 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 14, 2023-10-20T03:36:16.037 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.037 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.037 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.037 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.037 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.038 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.265757+0000", 2023-10-20T03:36:16.038 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.038 INFO:teuthology.orchestra.run.smithi033.stdout: "osd": 15, 2023-10-20T03:36:16.038 INFO:teuthology.orchestra.run.smithi033.stdout: "down_stamp": "0.000000", 2023-10-20T03:36:16.039 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_probability": 0, 2023-10-20T03:36:16.039 INFO:teuthology.orchestra.run.smithi033.stdout: "laggy_interval": 0, 2023-10-20T03:36:16.039 INFO:teuthology.orchestra.run.smithi033.stdout: "features": 4540138322906710015, 2023-10-20T03:36:16.039 INFO:teuthology.orchestra.run.smithi033.stdout: "old_weight": 0, 2023-10-20T03:36:16.039 INFO:teuthology.orchestra.run.smithi033.stdout: "last_purged_snaps_scrub": "2023-10-20T03:34:58.317166+0000", 2023-10-20T03:36:16.039 INFO:teuthology.orchestra.run.smithi033.stdout: "dead_epoch": 0 2023-10-20T03:36:16.040 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:16.040 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-10-20T03:36:16.040 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap": [], 2023-10-20T03:36:16.040 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap_items": [ 2023-10-20T03:36:16.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.040 INFO:teuthology.orchestra.run.smithi033.stdout: "pgid": "3.6", 2023-10-20T03:36:16.041 INFO:teuthology.orchestra.run.smithi033.stdout: "mappings": [ 2023-10-20T03:36:16.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.041 INFO:teuthology.orchestra.run.smithi033.stdout: "from": 8, 2023-10-20T03:36:16.041 INFO:teuthology.orchestra.run.smithi033.stdout: "to": 4 2023-10-20T03:36:16.041 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:16.041 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:16.042 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:16.042 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-10-20T03:36:16.042 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_upmap_primaries": [], 2023-10-20T03:36:16.042 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_temp": [], 2023-10-20T03:36:16.042 INFO:teuthology.orchestra.run.smithi033.stdout: "primary_temp": [], 2023-10-20T03:36:16.042 INFO:teuthology.orchestra.run.smithi033.stdout: "blocklist": {}, 2023-10-20T03:36:16.043 INFO:teuthology.orchestra.run.smithi033.stdout: "range_blocklist": {}, 2023-10-20T03:36:16.043 INFO:teuthology.orchestra.run.smithi033.stdout: "erasure_code_profiles": { 2023-10-20T03:36:16.043 INFO:teuthology.orchestra.run.smithi033.stdout: "default": { 2023-10-20T03:36:16.043 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-failure-domain": "osd", 2023-10-20T03:36:16.043 INFO:teuthology.orchestra.run.smithi033.stdout: "k": "2", 2023-10-20T03:36:16.043 INFO:teuthology.orchestra.run.smithi033.stdout: "m": "1", 2023-10-20T03:36:16.044 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin": "jerasure", 2023-10-20T03:36:16.044 INFO:teuthology.orchestra.run.smithi033.stdout: "technique": "reed_sol_van" 2023-10-20T03:36:16.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.044 INFO:teuthology.orchestra.run.smithi033.stdout: "shecprofile": { 2023-10-20T03:36:16.044 INFO:teuthology.orchestra.run.smithi033.stdout: "c": "2", 2023-10-20T03:36:16.044 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-device-class": "", 2023-10-20T03:36:16.045 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-failure-domain": "osd", 2023-10-20T03:36:16.045 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-root": "default", 2023-10-20T03:36:16.045 INFO:teuthology.orchestra.run.smithi033.stdout: "k": "4", 2023-10-20T03:36:16.045 INFO:teuthology.orchestra.run.smithi033.stdout: "m": "3", 2023-10-20T03:36:16.045 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "shecprofile", 2023-10-20T03:36:16.046 INFO:teuthology.orchestra.run.smithi033.stdout: "plugin": "shec", 2023-10-20T03:36:16.046 INFO:teuthology.orchestra.run.smithi033.stdout: "technique": "multiple" 2023-10-20T03:36:16.046 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:16.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.046 INFO:teuthology.orchestra.run.smithi033.stdout: "removed_snaps_queue": [ 2023-10-20T03:36:16.046 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.047 INFO:teuthology.orchestra.run.smithi033.stdout: "pool": 3, 2023-10-20T03:36:16.047 INFO:teuthology.orchestra.run.smithi033.stdout: "snaps": [ 2023-10-20T03:36:16.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.047 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 2, 2023-10-20T03:36:16.047 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 3 2023-10-20T03:36:16.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.048 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 6, 2023-10-20T03:36:16.048 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 2 2023-10-20T03:36:16.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.048 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.048 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 9, 2023-10-20T03:36:16.049 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 5 2023-10-20T03:36:16.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.049 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.049 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 15, 2023-10-20T03:36:16.049 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:36:16.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.050 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 22, 2023-10-20T03:36:16.050 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:36:16.050 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.050 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.050 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 25, 2023-10-20T03:36:16.051 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:36:16.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.051 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.051 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 28, 2023-10-20T03:36:16.051 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 3 2023-10-20T03:36:16.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.052 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 33, 2023-10-20T03:36:16.052 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:36:16.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.052 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.052 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 35, 2023-10-20T03:36:16.053 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:36:16.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.053 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.053 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 38, 2023-10-20T03:36:16.053 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 3 2023-10-20T03:36:16.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.054 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 43, 2023-10-20T03:36:16.054 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:36:16.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.054 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 46, 2023-10-20T03:36:16.055 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:36:16.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.055 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.055 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 48, 2023-10-20T03:36:16.055 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 2 2023-10-20T03:36:16.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.056 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 51, 2023-10-20T03:36:16.056 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:36:16.056 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:16.056 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:16.056 INFO:teuthology.orchestra.run.smithi033.stdout: "begin": 54, 2023-10-20T03:36:16.056 INFO:teuthology.orchestra.run.smithi033.stdout: "length": 1 2023-10-20T03:36:16.057 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:16.057 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:16.057 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:16.057 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-10-20T03:36:16.057 INFO:teuthology.orchestra.run.smithi033.stdout: "new_removed_snaps": [], 2023-10-20T03:36:16.057 INFO:teuthology.orchestra.run.smithi033.stdout: "new_purged_snaps": [], 2023-10-20T03:36:16.058 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_node_flags": {}, 2023-10-20T03:36:16.058 INFO:teuthology.orchestra.run.smithi033.stdout: "device_class_flags": {}, 2023-10-20T03:36:16.058 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode": { 2023-10-20T03:36:16.058 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_enabled": false, 2023-10-20T03:36:16.058 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_bucket_count": 0, 2023-10-20T03:36:16.059 INFO:teuthology.orchestra.run.smithi033.stdout: "degraded_stretch_mode": 0, 2023-10-20T03:36:16.059 INFO:teuthology.orchestra.run.smithi033.stdout: "recovering_stretch_mode": 0, 2023-10-20T03:36:16.059 INFO:teuthology.orchestra.run.smithi033.stdout: "stretch_mode_bucket": 0 2023-10-20T03:36:16.059 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:16.059 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:16.061 INFO:tasks.ceph.osd.9.smithi040.stderr:daemon-helper: command crashed with signal 6 2023-10-20T03:36:16.062 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:16.062 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:16.063 INFO:tasks.thrashosds.thrasher:j is {'epoch': 93, 'fsid': 'b920f165-e9e7-4c3e-8d7b-cfdc002ae589', 'created': '2023-10-20T03:34:47.883622+0000', 'modified': '2023-10-20T03:36:15.126462+0000', 'last_up_change': '2023-10-20T03:36:11.089758+0000', 'last_in_change': '2023-10-20T03:35:35.398012+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-20T03:35:00.991869+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9394655227661133, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-20T03:35:03.136216+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 5.949999809265137, 'score_stable': 5.949999809265137, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 3.75, 'raw_score_stable': 3.75, 'primary_affinity_weighted': 0.9394655227661133, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2023-10-20T03:35:12.214840+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 7, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 58, 'snap_epoch': 92, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'shecprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '4872451c-1bc7-4f21-a95f-986fa428ea65', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.072998046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 55, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6824', 'nonce': 121544157}, {'type': 'v1', 'addr': '172.21.15.33:6825', 'nonce': 121544157}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6826', 'nonce': 121544157}, {'type': 'v1', 'addr': '172.21.15.33:6827', 'nonce': 121544157}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6830', 'nonce': 121544157}, {'type': 'v1', 'addr': '172.21.15.33:6831', 'nonce': 121544157}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6828', 'nonce': 121544157}, {'type': 'v1', 'addr': '172.21.15.33:6829', 'nonce': 121544157}]}, 'public_addr': '172.21.15.33:6825/121544157', 'cluster_addr': '172.21.15.33:6827/121544157', 'heartbeat_back_addr': '172.21.15.33:6831/121544157', 'heartbeat_front_addr': '172.21.15.33:6829/121544157', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0b570426-e165-4caa-a914-e1bed7d805fc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6808', 'nonce': 562043184}, {'type': 'v1', 'addr': '172.21.15.40:6809', 'nonce': 562043184}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6810', 'nonce': 562043184}, {'type': 'v1', 'addr': '172.21.15.40:6811', 'nonce': 562043184}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6818', 'nonce': 562043184}, {'type': 'v1', 'addr': '172.21.15.40:6820', 'nonce': 562043184}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6813', 'nonce': 562043184}, {'type': 'v1', 'addr': '172.21.15.40:6815', 'nonce': 562043184}]}, 'public_addr': '172.21.15.40:6809/562043184', 'cluster_addr': '172.21.15.40:6811/562043184', 'heartbeat_back_addr': '172.21.15.40:6820/562043184', 'heartbeat_front_addr': '172.21.15.40:6815/562043184', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'e0184c10-844b-4bcb-adaf-a0d407b708ee', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6824', 'nonce': 3785405088}, {'type': 'v1', 'addr': '172.21.15.64:6825', 'nonce': 3785405088}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6826', 'nonce': 3785405088}, {'type': 'v1', 'addr': '172.21.15.64:6827', 'nonce': 3785405088}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6830', 'nonce': 3785405088}, {'type': 'v1', 'addr': '172.21.15.64:6831', 'nonce': 3785405088}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6828', 'nonce': 3785405088}, {'type': 'v1', 'addr': '172.21.15.64:6829', 'nonce': 3785405088}]}, 'public_addr': '172.21.15.64:6825/3785405088', 'cluster_addr': '172.21.15.64:6827/3785405088', 'heartbeat_back_addr': '172.21.15.64:6831/3785405088', 'heartbeat_front_addr': '172.21.15.64:6829/3785405088', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8120023b-1d48-411d-a642-2529a697b5c9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9584503173828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6818', 'nonce': 1426674535}, {'type': 'v1', 'addr': '172.21.15.170:6820', 'nonce': 1426674535}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6821', 'nonce': 1426674535}, {'type': 'v1', 'addr': '172.21.15.170:6823', 'nonce': 1426674535}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6829', 'nonce': 1426674535}, {'type': 'v1', 'addr': '172.21.15.170:6831', 'nonce': 1426674535}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6825', 'nonce': 1426674535}, {'type': 'v1', 'addr': '172.21.15.170:6827', 'nonce': 1426674535}]}, 'public_addr': '172.21.15.170:6820/1426674535', 'cluster_addr': '172.21.15.170:6823/1426674535', 'heartbeat_back_addr': '172.21.15.170:6831/1426674535', 'heartbeat_front_addr': '172.21.15.170:6827/1426674535', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '549b3b39-41d5-4d9b-932e-a7b3e8cc0684', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 55, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6808', 'nonce': 3186493546}, {'type': 'v1', 'addr': '172.21.15.33:6809', 'nonce': 3186493546}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6810', 'nonce': 3186493546}, {'type': 'v1', 'addr': '172.21.15.33:6811', 'nonce': 3186493546}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6814', 'nonce': 3186493546}, {'type': 'v1', 'addr': '172.21.15.33:6815', 'nonce': 3186493546}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6812', 'nonce': 3186493546}, {'type': 'v1', 'addr': '172.21.15.33:6813', 'nonce': 3186493546}]}, 'public_addr': '172.21.15.33:6809/3186493546', 'cluster_addr': '172.21.15.33:6811/3186493546', 'heartbeat_back_addr': '172.21.15.33:6815/3186493546', 'heartbeat_front_addr': '172.21.15.33:6813/3186493546', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'a99ea3db-6433-4312-83d1-64ba69f028a4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6814', 'nonce': 1211670976}, {'type': 'v1', 'addr': '172.21.15.40:6817', 'nonce': 1211670976}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6821', 'nonce': 1211670976}, {'type': 'v1', 'addr': '172.21.15.40:6823', 'nonce': 1211670976}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6829', 'nonce': 1211670976}, {'type': 'v1', 'addr': '172.21.15.40:6831', 'nonce': 1211670976}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6825', 'nonce': 1211670976}, {'type': 'v1', 'addr': '172.21.15.40:6827', 'nonce': 1211670976}]}, 'public_addr': '172.21.15.40:6817/1211670976', 'cluster_addr': '172.21.15.40:6823/1211670976', 'heartbeat_back_addr': '172.21.15.40:6831/1211670976', 'heartbeat_front_addr': '172.21.15.40:6827/1211670976', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd2b38063-49eb-4f56-bb8d-411419b71b9b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 22, 'last_clean_end': 28, 'up_from': 89, 'up_thru': 92, 'down_at': 29, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 2043848066}, {'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 2043848066}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 2043848066}, {'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 2043848066}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 2043848066}, {'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 2043848066}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 2043848066}, {'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 2043848066}]}, 'public_addr': '172.21.15.64:6809/2043848066', 'cluster_addr': '172.21.15.64:6811/2043848066', 'heartbeat_back_addr': '172.21.15.64:6815/2043848066', 'heartbeat_front_addr': '172.21.15.64:6813/2043848066', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '80e34f64-35dc-4a6d-9f7f-c2263c78350d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6810', 'nonce': 2804292733}, {'type': 'v1', 'addr': '172.21.15.170:6811', 'nonce': 2804292733}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6812', 'nonce': 2804292733}, {'type': 'v1', 'addr': '172.21.15.170:6813', 'nonce': 2804292733}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6816', 'nonce': 2804292733}, {'type': 'v1', 'addr': '172.21.15.170:6817', 'nonce': 2804292733}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6814', 'nonce': 2804292733}, {'type': 'v1', 'addr': '172.21.15.170:6815', 'nonce': 2804292733}]}, 'public_addr': '172.21.15.170:6811/2804292733', 'cluster_addr': '172.21.15.170:6813/2804292733', 'heartbeat_back_addr': '172.21.15.170:6817/2804292733', 'heartbeat_front_addr': '172.21.15.170:6815/2804292733', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '1cd10625-66a8-4488-b1e5-378cb0062107', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6800', 'nonce': 1638010589}, {'type': 'v1', 'addr': '172.21.15.33:6801', 'nonce': 1638010589}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6802', 'nonce': 1638010589}, {'type': 'v1', 'addr': '172.21.15.33:6803', 'nonce': 1638010589}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6806', 'nonce': 1638010589}, {'type': 'v1', 'addr': '172.21.15.33:6807', 'nonce': 1638010589}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6804', 'nonce': 1638010589}, {'type': 'v1', 'addr': '172.21.15.33:6805', 'nonce': 1638010589}]}, 'public_addr': '172.21.15.33:6801/1638010589', 'cluster_addr': '172.21.15.33:6803/1638010589', 'heartbeat_back_addr': '172.21.15.33:6807/1638010589', 'heartbeat_front_addr': '172.21.15.33:6805/1638010589', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6800', 'nonce': 2173797306}, {'type': 'v1', 'addr': '172.21.15.40:6801', 'nonce': 2173797306}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6802', 'nonce': 2173797306}, {'type': 'v1', 'addr': '172.21.15.40:6803', 'nonce': 2173797306}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6806', 'nonce': 2173797306}, {'type': 'v1', 'addr': '172.21.15.40:6807', 'nonce': 2173797306}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6804', 'nonce': 2173797306}, {'type': 'v1', 'addr': '172.21.15.40:6805', 'nonce': 2173797306}]}, 'public_addr': '172.21.15.40:6801/2173797306', 'cluster_addr': '172.21.15.40:6803/2173797306', 'heartbeat_back_addr': '172.21.15.40:6807/2173797306', 'heartbeat_front_addr': '172.21.15.40:6805/2173797306', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '42a67bcc-5c07-499d-8902-08c2f15e7fbb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 56, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 91701057}, {'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 91701057}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 91701057}, {'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 91701057}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 91701057}, {'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 91701057}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 91701057}, {'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 91701057}]}, 'public_addr': '172.21.15.64:6801/91701057', 'cluster_addr': '172.21.15.64:6803/91701057', 'heartbeat_back_addr': '172.21.15.64:6807/91701057', 'heartbeat_front_addr': '172.21.15.64:6805/91701057', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ab1db647-aada-4deb-abdc-0a1baff548db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6802', 'nonce': 1717984751}, {'type': 'v1', 'addr': '172.21.15.170:6803', 'nonce': 1717984751}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6804', 'nonce': 1717984751}, {'type': 'v1', 'addr': '172.21.15.170:6805', 'nonce': 1717984751}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6808', 'nonce': 1717984751}, {'type': 'v1', 'addr': '172.21.15.170:6809', 'nonce': 1717984751}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6806', 'nonce': 1717984751}, {'type': 'v1', 'addr': '172.21.15.170:6807', 'nonce': 1717984751}]}, 'public_addr': '172.21.15.170:6803/1717984751', 'cluster_addr': '172.21.15.170:6805/1717984751', 'heartbeat_back_addr': '172.21.15.170:6809/1717984751', 'heartbeat_front_addr': '172.21.15.170:6807/1717984751', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': 'b20cee74-2cce-4153-97c2-56e9db40c712', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6816', 'nonce': 3797427065}, {'type': 'v1', 'addr': '172.21.15.33:6817', 'nonce': 3797427065}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6818', 'nonce': 3797427065}, {'type': 'v1', 'addr': '172.21.15.33:6819', 'nonce': 3797427065}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6822', 'nonce': 3797427065}, {'type': 'v1', 'addr': '172.21.15.33:6823', 'nonce': 3797427065}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.33:6820', 'nonce': 3797427065}, {'type': 'v1', 'addr': '172.21.15.33:6821', 'nonce': 3797427065}]}, 'public_addr': '172.21.15.33:6817/3797427065', 'cluster_addr': '172.21.15.33:6819/3797427065', 'heartbeat_back_addr': '172.21.15.33:6823/3797427065', 'heartbeat_front_addr': '172.21.15.33:6821/3797427065', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '0974cd04-c690-4cba-ac05-5d404493dd1d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6812', 'nonce': 3041424233}, {'type': 'v1', 'addr': '172.21.15.40:6816', 'nonce': 3041424233}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6819', 'nonce': 3041424233}, {'type': 'v1', 'addr': '172.21.15.40:6822', 'nonce': 3041424233}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6828', 'nonce': 3041424233}, {'type': 'v1', 'addr': '172.21.15.40:6830', 'nonce': 3041424233}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6824', 'nonce': 3041424233}, {'type': 'v1', 'addr': '172.21.15.40:6826', 'nonce': 3041424233}]}, 'public_addr': '172.21.15.40:6816/3041424233', 'cluster_addr': '172.21.15.40:6822/3041424233', 'heartbeat_back_addr': '172.21.15.40:6830/3041424233', 'heartbeat_front_addr': '172.21.15.40:6826/3041424233', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '3bf53124-13b6-4e5c-9868-2382719ffe25', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6816', 'nonce': 2940007286}, {'type': 'v1', 'addr': '172.21.15.64:6817', 'nonce': 2940007286}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6818', 'nonce': 2940007286}, {'type': 'v1', 'addr': '172.21.15.64:6819', 'nonce': 2940007286}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6822', 'nonce': 2940007286}, {'type': 'v1', 'addr': '172.21.15.64:6823', 'nonce': 2940007286}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6820', 'nonce': 2940007286}, {'type': 'v1', 'addr': '172.21.15.64:6821', 'nonce': 2940007286}]}, 'public_addr': '172.21.15.64:6817/2940007286', 'cluster_addr': '172.21.15.64:6819/2940007286', 'heartbeat_back_addr': '172.21.15.64:6823/2940007286', 'heartbeat_front_addr': '172.21.15.64:6821/2940007286', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': 'b0b5ffe5-a25b-412f-818f-b60907c0089b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6819', 'nonce': 826332016}, {'type': 'v1', 'addr': '172.21.15.170:6822', 'nonce': 826332016}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6824', 'nonce': 826332016}, {'type': 'v1', 'addr': '172.21.15.170:6826', 'nonce': 826332016}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6832', 'nonce': 826332016}, {'type': 'v1', 'addr': '172.21.15.170:6833', 'nonce': 826332016}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6828', 'nonce': 826332016}, {'type': 'v1', 'addr': '172.21.15.170:6830', 'nonce': 826332016}]}, 'public_addr': '172.21.15.170:6822/826332016', 'cluster_addr': '172.21.15.170:6826/826332016', 'heartbeat_back_addr': '172.21.15.170:6833/826332016', 'heartbeat_front_addr': '172.21.15.170:6830/826332016', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.204996+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:36:11.884878+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:36:11.632626+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.175698+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:36:03.397982+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:36:03.463084+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:36:10.320469+0000', 'dead_epoch': 29}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:35:58.321221+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:36:10.321913+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:35:59.745876+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:36:02.082859+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:36:03.991453+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-10-20T03:36:05.214755+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:36:01.467282+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.265757+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-10-20T03:34:58.317166+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.6', 'mappings': [{'from': 8, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'shecprofile': {'c': '2', 'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'k': '4', 'm': '3', 'name': 'shecprofile', 'plugin': 'shec', 'technique': 'multiple'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 3}, {'begin': 6, 'length': 2}, {'begin': 9, 'length': 5}, {'begin': 15, 'length': 1}, {'begin': 22, 'length': 1}, {'begin': 25, 'length': 1}, {'begin': 28, 'length': 3}, {'begin': 33, 'length': 1}, {'begin': 35, 'length': 1}, {'begin': 38, 'length': 3}, {'begin': 43, 'length': 1}, {'begin': 46, 'length': 1}, {'begin': 48, 'length': 2}, {'begin': 51, 'length': 1}, {'begin': 54, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-10-20T03:36:16.064 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2023-10-20T03:36:16.064 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:16.056+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:16.064 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-10-20T03:36:16.162 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:36:16.159+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:36:16.168 INFO:tasks.rados.rados.0.smithi170.stdout:324: done (5 left) 2023-10-20T03:36:16.168 INFO:tasks.rados.rados.0.smithi170.stdout:328: done (4 left) 2023-10-20T03:36:16.168 INFO:tasks.rados.rados.0.smithi170.stdout:330: done (3 left) 2023-10-20T03:36:16.168 INFO:tasks.rados.rados.0.smithi170.stdout:331: done (2 left) 2023-10-20T03:36:16.168 INFO:tasks.rados.rados.0.smithi170.stdout:332: done (1 left) 2023-10-20T03:36:16.169 INFO:tasks.rados.rados.0.smithi170.stdout:333: done (0 left) 2023-10-20T03:36:16.169 INFO:tasks.rados.rados.0.smithi170.stdout:334: snap_remove snap 30 2023-10-20T03:36:16.199 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:16.199 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:16.213 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-10-20T03:36:16.263 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:16.260+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:16.348 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:16.348 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:16.362 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-10-20T03:36:16.363 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:36:16.359+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:36:16.465 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:16.461+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:16.480 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:16.480 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:16.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-10-20T03:36:16.566 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:16.564+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:16.621 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:16.621 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:16.636 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-10-20T03:36:16.666 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:36:16.686 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:16.684+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:16.766 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-10-20T03:36:16.775 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:16.775 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:16.791 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-10-20T03:36:16.833 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:36:16.834 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:36:16.854 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:36:16.854 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:36:16.869 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:16.864+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:16.874 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:36:16.874 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:36:16.894 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:36:16.895 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:36:16.911 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:36:16.911 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:36:16.920 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:16.920 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:36:16.921 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:36:16.921 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:36:16.921 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:16.933 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:36:16.933 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:36:16.935 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-10-20T03:36:16.953 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2023-10-20T03:36:16.953 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-10-20T03:36:16.970 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:16.967+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:16.973 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:36:16.973 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:36:16.990 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:36:16.990 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:36:16.991 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.9 2023-10-20T03:36:17.012 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:36:17.012 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:36:17.033 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:36:17.033 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:36:17.050 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:36:17.051 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:36:17.070 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:17.064+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:17.072 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:17.072 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:36:17.072 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:36:17.073 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:36:17.073 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.073 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s5 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:36:17.073 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.356693+0000", 2023-10-20T03:36:17.073 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 51.713395294999998, 2023-10-20T03:36:17.074 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010267518, 2023-10-20T03:36:17.074 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.074 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.074 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.074 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.075 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.075 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356693+0000", 2023-10-20T03:36:17.075 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.076 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.076 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356693+0000", 2023-10-20T03:36:17.076 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.076 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.076 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.077 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.077 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356694+0000", 2023-10-20T03:36:17.077 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5000000000000002e-07 2023-10-20T03:36:17.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.077 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.078 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.078 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357174+0000", 2023-10-20T03:36:17.078 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00047994599999999998 2023-10-20T03:36:17.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.078 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.079 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.079 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357194+0000", 2023-10-20T03:36:17.079 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.978e-05 2023-10-20T03:36:17.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.079 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.080 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.080 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357209+0000", 2023-10-20T03:36:17.080 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5506e-05 2023-10-20T03:36:17.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.081 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.081 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357336+0000", 2023-10-20T03:36:17.081 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012640000000000001 2023-10-20T03:36:17.081 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.081 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.082 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.082 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357353+0000", 2023-10-20T03:36:17.082 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6867e-05 2023-10-20T03:36:17.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.082 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.083 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.083 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.366907+0000", 2023-10-20T03:36:17.083 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.009554412 2023-10-20T03:36:17.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.084 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.084 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.084 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.366961+0000", 2023-10-20T03:36:17.084 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3956999999999999e-05 2023-10-20T03:36:17.084 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.085 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.085 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.085 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.085 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:331 3.cs0 3:3382ea17:::smithi17039300-35:head [rollback a] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:17.086 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.371212+0000", 2023-10-20T03:36:17.086 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 50.698876908999999, 2023-10-20T03:36:17.086 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010542586, 2023-10-20T03:36:17.086 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.086 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:17.087 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:17.087 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:17.087 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:17.087 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 331 2023-10-20T03:36:17.087 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.088 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.088 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.088 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371212+0000", 2023-10-20T03:36:17.088 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.089 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.089 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.089 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.089 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371212+0000", 2023-10-20T03:36:17.089 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.090 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.090 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.090 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371213+0000", 2023-10-20T03:36:17.090 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.1999999999999998e-07 2023-10-20T03:36:17.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.091 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.091 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.091 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371219+0000", 2023-10-20T03:36:17.091 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2600000000000002e-06 2023-10-20T03:36:17.092 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.092 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.092 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.092 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371233+0000", 2023-10-20T03:36:17.092 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3816e-05 2023-10-20T03:36:17.092 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.093 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.093 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371241+0000", 2023-10-20T03:36:17.093 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5569999999999996e-06 2023-10-20T03:36:17.094 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.094 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.094 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.094 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.372719+0000", 2023-10-20T03:36:17.094 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014780539999999999 2023-10-20T03:36:17.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.095 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.095 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:17.095 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.372777+0000", 2023-10-20T03:36:17.095 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.7485999999999998e-05 2023-10-20T03:36:17.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.096 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.096 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.096 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.373138+0000", 2023-10-20T03:36:17.096 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000361679 2023-10-20T03:36:17.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.097 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.097 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.097 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.376244+0000", 2023-10-20T03:36:17.097 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.003105652 2023-10-20T03:36:17.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.098 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:17.098 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381631+0000", 2023-10-20T03:36:17.098 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053871179999999998 2023-10-20T03:36:17.099 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.099 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.099 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.099 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381754+0000", 2023-10-20T03:36:17.099 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000123144 2023-10-20T03:36:17.100 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.100 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.100 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.100 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.101 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:332 3.cs0 3:3382ea17:::smithi17039300-35:head [append 0~0] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:17.101 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.371334+0000", 2023-10-20T03:36:17.101 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 50.698754192000003, 2023-10-20T03:36:17.101 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010711231, 2023-10-20T03:36:17.101 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.102 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:17.102 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:17.102 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:17.102 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:17.103 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 332 2023-10-20T03:36:17.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.103 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.103 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.103 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.104 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371334+0000", 2023-10-20T03:36:17.104 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.104 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.104 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.104 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.104 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371334+0000", 2023-10-20T03:36:17.105 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.105 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.105 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371335+0000", 2023-10-20T03:36:17.106 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.47e-07 2023-10-20T03:36:17.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.106 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.106 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.106 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371340+0000", 2023-10-20T03:36:17.107 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5800000000000002e-06 2023-10-20T03:36:17.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.107 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.107 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.107 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371366+0000", 2023-10-20T03:36:17.108 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5639e-05 2023-10-20T03:36:17.108 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.108 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.108 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.108 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.371379+0000", 2023-10-20T03:36:17.109 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.346e-05 2023-10-20T03:36:17.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.109 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.109 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.110 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.373793+0000", 2023-10-20T03:36:17.110 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024143049999999998 2023-10-20T03:36:17.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.110 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.110 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:17.111 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.373855+0000", 2023-10-20T03:36:17.111 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1277000000000003e-05 2023-10-20T03:36:17.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.111 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.111 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.112 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.374199+0000", 2023-10-20T03:36:17.112 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00034445500000000002 2023-10-20T03:36:17.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.112 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.112 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.113 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.376260+0000", 2023-10-20T03:36:17.113 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0020611050000000001 2023-10-20T03:36:17.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.113 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.113 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:17.114 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.381945+0000", 2023-10-20T03:36:17.114 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056847690000000001 2023-10-20T03:36:17.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.114 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.115 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.382046+0000", 2023-10-20T03:36:17.115 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000100794 2023-10-20T03:36:17.115 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.115 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.116 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.116 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.116 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:17.116 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.387979+0000", 2023-10-20T03:36:17.117 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 50.682109951999998, 2023-10-20T03:36:17.117 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018548599999999998, 2023-10-20T03:36:17.117 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.117 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.117 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.118 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.118 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.118 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387979+0000", 2023-10-20T03:36:17.118 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.119 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.119 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.119 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387979+0000", 2023-10-20T03:36:17.119 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.119 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.120 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.120 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.120 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387979+0000", 2023-10-20T03:36:17.120 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6000000000000003e-07 2023-10-20T03:36:17.120 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.121 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.121 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388153+0000", 2023-10-20T03:36:17.121 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017364600000000001 2023-10-20T03:36:17.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.121 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.122 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.122 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388173+0000", 2023-10-20T03:36:17.122 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9961000000000001e-05 2023-10-20T03:36:17.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.123 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.123 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388186+0000", 2023-10-20T03:36:17.123 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2403e-05 2023-10-20T03:36:17.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.123 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.124 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.124 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388293+0000", 2023-10-20T03:36:17.124 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010739300000000001 2023-10-20T03:36:17.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.124 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.125 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.125 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388307+0000", 2023-10-20T03:36:17.125 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3773e-05 2023-10-20T03:36:17.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.125 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.126 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.126 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406469+0000", 2023-10-20T03:36:17.126 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018162098000000002 2023-10-20T03:36:17.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.126 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.127 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.127 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406527+0000", 2023-10-20T03:36:17.127 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8566000000000001e-05 2023-10-20T03:36:17.127 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.127 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.128 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.128 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:17.129 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.394884+0000", 2023-10-20T03:36:17.129 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 50.675204854, 2023-10-20T03:36:17.129 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011674960999999999, 2023-10-20T03:36:17.129 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.129 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.130 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.130 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.130 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.130 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394884+0000", 2023-10-20T03:36:17.131 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.131 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.131 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.131 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394884+0000", 2023-10-20T03:36:17.132 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.132 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.132 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.132 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.132 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394886+0000", 2023-10-20T03:36:17.133 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0269999999999998e-06 2023-10-20T03:36:17.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.133 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.133 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.133 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395153+0000", 2023-10-20T03:36:17.134 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026702300000000002 2023-10-20T03:36:17.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.134 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.134 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.134 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395187+0000", 2023-10-20T03:36:17.135 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3738999999999997e-05 2023-10-20T03:36:17.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.135 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.135 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.135 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395211+0000", 2023-10-20T03:36:17.136 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4859e-05 2023-10-20T03:36:17.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.136 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.136 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395301+0000", 2023-10-20T03:36:17.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9553000000000001e-05 2023-10-20T03:36:17.137 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.137 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.137 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.137 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.395310+0000", 2023-10-20T03:36:17.137 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9439999999999994e-06 2023-10-20T03:36:17.138 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.138 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.138 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.138 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406535+0000", 2023-10-20T03:36:17.138 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011225153999999999 2023-10-20T03:36:17.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.139 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.139 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.139 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406559+0000", 2023-10-20T03:36:17.139 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3662000000000001e-05 2023-10-20T03:36:17.140 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.140 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.140 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.140 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.141 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs0 48/47 ECSubWrite(tid=115, reqid=osd.13.0:115, at_version=48'27, trim_to=0'0, roll_forward_to=0'0))", 2023-10-20T03:36:17.141 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.389413+0000", 2023-10-20T03:36:17.141 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 46.680675391999998, 2023-10-20T03:36:17.141 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0097928949999999994, 2023-10-20T03:36:17.141 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.141 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.142 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.142 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.142 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.142 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389413+0000", 2023-10-20T03:36:17.142 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.143 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.143 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.143 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389413+0000", 2023-10-20T03:36:17.143 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.144 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.144 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.144 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389416+0000", 2023-10-20T03:36:17.144 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2170000000000001e-06 2023-10-20T03:36:17.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.145 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.145 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.145 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389474+0000", 2023-10-20T03:36:17.145 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8931999999999998e-05 2023-10-20T03:36:17.145 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.146 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.146 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389502+0000", 2023-10-20T03:36:17.146 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7538999999999998e-05 2023-10-20T03:36:17.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.147 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.147 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.147 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389531+0000", 2023-10-20T03:36:17.147 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9172000000000001e-05 2023-10-20T03:36:17.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.148 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.148 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.148 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389702+0000", 2023-10-20T03:36:17.148 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017064000000000001 2023-10-20T03:36:17.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.149 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.149 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.149 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389767+0000", 2023-10-20T03:36:17.149 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5297999999999997e-05 2023-10-20T03:36:17.149 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.150 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.150 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.150 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.399101+0000", 2023-10-20T03:36:17.150 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093336600000000006 2023-10-20T03:36:17.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.151 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.151 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.399206+0000", 2023-10-20T03:36:17.151 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000105437 2023-10-20T03:36:17.151 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.152 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.152 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.152 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.152 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.153 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:36:17.153 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.392018+0000", 2023-10-20T03:36:17.153 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 46.678070912999999, 2023-10-20T03:36:17.153 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037305552999999998, 2023-10-20T03:36:17.153 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.154 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.154 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.154 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.154 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.155 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392018+0000", 2023-10-20T03:36:17.155 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.155 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.156 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.156 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392018+0000", 2023-10-20T03:36:17.156 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.156 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.157 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.157 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392020+0000", 2023-10-20T03:36:17.157 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2029999999999999e-06 2023-10-20T03:36:17.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.157 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.158 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.158 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392961+0000", 2023-10-20T03:36:17.158 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00094143000000000004 2023-10-20T03:36:17.158 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.159 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.159 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.159 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392993+0000", 2023-10-20T03:36:17.159 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.1795999999999998e-05 2023-10-20T03:36:17.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.160 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.160 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.160 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393083+0000", 2023-10-20T03:36:17.160 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9584000000000002e-05 2023-10-20T03:36:17.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.161 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.161 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.161 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393389+0000", 2023-10-20T03:36:17.161 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00030593699999999999 2023-10-20T03:36:17.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.162 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.162 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.162 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393416+0000", 2023-10-20T03:36:17.162 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7639000000000001e-05 2023-10-20T03:36:17.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.163 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.163 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.163 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429229+0000", 2023-10-20T03:36:17.163 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.035812384000000003 2023-10-20T03:36:17.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.164 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.164 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.164 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429323+0000", 2023-10-20T03:36:17.164 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4580000000000006e-05 2023-10-20T03:36:17.165 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.165 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.165 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.165 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.165 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s3 49/31 ECSubWrite(tid=25, reqid=osd.10.0:24, at_version=0'0, trim_to=0'0, roll_forward_to=49'15))", 2023-10-20T03:36:17.166 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.408468+0000", 2023-10-20T03:36:17.166 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 45.661620206000002, 2023-10-20T03:36:17.166 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014029283, 2023-10-20T03:36:17.166 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.167 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.167 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.167 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.167 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.167 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408468+0000", 2023-10-20T03:36:17.168 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.168 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.168 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.168 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408468+0000", 2023-10-20T03:36:17.169 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.169 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.169 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.169 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408470+0000", 2023-10-20T03:36:17.170 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1680000000000001e-06 2023-10-20T03:36:17.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.170 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.170 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.170 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408486+0000", 2023-10-20T03:36:17.171 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6209999999999999e-05 2023-10-20T03:36:17.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.171 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.171 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.171 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408502+0000", 2023-10-20T03:36:17.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6351e-05 2023-10-20T03:36:17.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.172 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.172 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.172 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408514+0000", 2023-10-20T03:36:17.172 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1364e-05 2023-10-20T03:36:17.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.173 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.173 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.173 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408748+0000", 2023-10-20T03:36:17.173 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000234088 2023-10-20T03:36:17.174 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.174 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.174 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.408763+0000", 2023-10-20T03:36:17.175 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5586999999999998e-05 2023-10-20T03:36:17.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.175 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.175 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.175 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.422444+0000", 2023-10-20T03:36:17.176 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013680701999999999 2023-10-20T03:36:17.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.176 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.176 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.176 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.422498+0000", 2023-10-20T03:36:17.176 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3813000000000001e-05 2023-10-20T03:36:17.177 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.177 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.177 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.177 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.178 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.fs0 53/47 ECSubWrite(tid=57, reqid=client.4570.0:408, at_version=53'31, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:17.178 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:35.448532+0000", 2023-10-20T03:36:17.178 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.621556796999997, 2023-10-20T03:36:17.178 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012712822, 2023-10-20T03:36:17.178 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.179 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.179 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.179 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.179 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.179 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.448532+0000", 2023-10-20T03:36:17.179 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.180 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.180 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.180 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.448532+0000", 2023-10-20T03:36:17.180 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.180 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.181 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.181 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.448532+0000", 2023-10-20T03:36:17.181 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2799999999999996e-07 2023-10-20T03:36:17.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.182 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.182 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.182 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449112+0000", 2023-10-20T03:36:17.182 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00057979400000000001 2023-10-20T03:36:17.182 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.183 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.183 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.183 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449124+0000", 2023-10-20T03:36:17.183 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1613999999999999e-05 2023-10-20T03:36:17.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.184 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.184 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.184 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449129+0000", 2023-10-20T03:36:17.184 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.6049999999999997e-06 2023-10-20T03:36:17.184 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.185 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.185 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.185 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449245+0000", 2023-10-20T03:36:17.185 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011568199999999999 2023-10-20T03:36:17.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.186 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.186 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.186 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.449261+0000", 2023-10-20T03:36:17.186 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6376999999999999e-05 2023-10-20T03:36:17.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.187 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.187 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.187 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.461172+0000", 2023-10-20T03:36:17.187 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01191096 2023-10-20T03:36:17.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.188 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.188 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.188 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.461245+0000", 2023-10-20T03:36:17.188 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2261999999999994e-05 2023-10-20T03:36:17.189 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.189 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.189 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.189 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.189 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.190 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.fs0 53/47 ECSubWrite(tid=58, reqid=client.4570.0:409, at_version=53'32, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:17.190 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:35.472456+0000", 2023-10-20T03:36:17.190 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.597632261999998, 2023-10-20T03:36:17.190 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011134987000000001, 2023-10-20T03:36:17.190 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.191 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.191 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.191 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.191 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472456+0000", 2023-10-20T03:36:17.192 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.192 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.192 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.192 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472456+0000", 2023-10-20T03:36:17.193 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.193 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.193 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.193 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.193 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472457+0000", 2023-10-20T03:36:17.194 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7999999999999996e-07 2023-10-20T03:36:17.194 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.194 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.194 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.194 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472728+0000", 2023-10-20T03:36:17.195 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00027096800000000001 2023-10-20T03:36:17.195 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.195 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.195 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.195 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472738+0000", 2023-10-20T03:36:17.196 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0495e-05 2023-10-20T03:36:17.196 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.196 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.196 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.196 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472744+0000", 2023-10-20T03:36:17.197 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.0410000000000003e-06 2023-10-20T03:36:17.197 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.197 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.197 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.197 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472867+0000", 2023-10-20T03:36:17.198 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012291400000000001 2023-10-20T03:36:17.198 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.198 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.198 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.198 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.472883+0000", 2023-10-20T03:36:17.199 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5628e-05 2023-10-20T03:36:17.199 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.199 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.199 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.199 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.483517+0000", 2023-10-20T03:36:17.200 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010634423 2023-10-20T03:36:17.200 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.200 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.200 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.200 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:35.483591+0000", 2023-10-20T03:36:17.200 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4037999999999998e-05 2023-10-20T03:36:17.201 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.201 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.201 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.201 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.201 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.202 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.bs3 60/56 ECSubWrite(tid=204, reqid=client.4570.0:470, at_version=60'24, trim_to=0'0, roll_forward_to=60'22))", 2023-10-20T03:36:17.202 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.923867+0000", 2023-10-20T03:36:17.202 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.146221287000003, 2023-10-20T03:36:17.202 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014718472999999999, 2023-10-20T03:36:17.203 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.203 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.203 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.203 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.203 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.204 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.923867+0000", 2023-10-20T03:36:17.204 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.204 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.204 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.204 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.205 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.923867+0000", 2023-10-20T03:36:17.205 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.205 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.205 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.205 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.206 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.923868+0000", 2023-10-20T03:36:17.206 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.063e-06 2023-10-20T03:36:17.206 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.206 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.206 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.207 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924029+0000", 2023-10-20T03:36:17.207 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000160927 2023-10-20T03:36:17.207 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.207 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.207 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.207 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924045+0000", 2023-10-20T03:36:17.208 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5874000000000001e-05 2023-10-20T03:36:17.208 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.208 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.208 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.209 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924054+0000", 2023-10-20T03:36:17.209 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7320000000000004e-06 2023-10-20T03:36:17.209 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.209 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.209 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.210 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924173+0000", 2023-10-20T03:36:17.210 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000118713 2023-10-20T03:36:17.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.210 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.211 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.211 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.924190+0000", 2023-10-20T03:36:17.211 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7280000000000001e-05 2023-10-20T03:36:17.211 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.211 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.212 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.212 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.938488+0000", 2023-10-20T03:36:17.212 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014298132 2023-10-20T03:36:17.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.212 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.213 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.213 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.938586+0000", 2023-10-20T03:36:17.213 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.7751999999999995e-05 2023-10-20T03:36:17.213 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.213 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.214 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.214 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.214 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.214 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s3 70/56 ECSubWrite(tid=57, reqid=client.4570.0:558, at_version=70'42, trim_to=0'0, roll_forward_to=70'40))", 2023-10-20T03:36:17.215 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:51.998575+0000", 2023-10-20T03:36:17.215 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.071513796000001, 2023-10-20T03:36:17.215 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013366552, 2023-10-20T03:36:17.215 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.215 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.215 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.216 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.216 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.216 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998575+0000", 2023-10-20T03:36:17.216 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.217 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.217 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.217 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998575+0000", 2023-10-20T03:36:17.217 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.217 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.218 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.218 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.218 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998577+0000", 2023-10-20T03:36:17.218 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.967e-06 2023-10-20T03:36:17.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.219 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.219 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.219 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998782+0000", 2023-10-20T03:36:17.219 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020551 2023-10-20T03:36:17.219 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.220 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.220 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.220 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998818+0000", 2023-10-20T03:36:17.220 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5879e-05 2023-10-20T03:36:17.220 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.221 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.221 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.221 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998836+0000", 2023-10-20T03:36:17.221 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7583000000000001e-05 2023-10-20T03:36:17.221 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.222 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.222 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.222 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998975+0000", 2023-10-20T03:36:17.222 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000139231 2023-10-20T03:36:17.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.223 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.223 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.223 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:51.998994+0000", 2023-10-20T03:36:17.223 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8936e-05 2023-10-20T03:36:17.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.224 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.224 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.224 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.011876+0000", 2023-10-20T03:36:17.224 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012881951000000001 2023-10-20T03:36:17.224 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.225 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.225 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.225 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.011941+0000", 2023-10-20T03:36:17.225 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.5494999999999995e-05 2023-10-20T03:36:17.226 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.226 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.226 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.226 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.226 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.227 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.bs3 70/56 ECSubWrite(tid=260, reqid=client.4570.0:561, at_version=70'36, trim_to=0'0, roll_forward_to=61'34))", 2023-10-20T03:36:17.227 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:52.022553+0000", 2023-10-20T03:36:17.227 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.047536125000001, 2023-10-20T03:36:17.227 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012554222, 2023-10-20T03:36:17.227 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.228 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.228 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.228 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.228 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.228 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.022553+0000", 2023-10-20T03:36:17.229 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.229 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.229 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.229 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.022553+0000", 2023-10-20T03:36:17.230 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.230 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.230 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.231 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.022553+0000", 2023-10-20T03:36:17.231 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8000000000000001e-07 2023-10-20T03:36:17.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.231 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.231 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.232 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.022966+0000", 2023-10-20T03:36:17.232 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00041324899999999997 2023-10-20T03:36:17.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.232 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.233 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.022988+0000", 2023-10-20T03:36:17.233 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1557999999999999e-05 2023-10-20T03:36:17.233 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.233 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.233 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.233 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.023003+0000", 2023-10-20T03:36:17.234 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5150000000000001e-05 2023-10-20T03:36:17.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.234 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.234 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.023112+0000", 2023-10-20T03:36:17.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000108798 2023-10-20T03:36:17.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.235 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.235 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.235 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.023128+0000", 2023-10-20T03:36:17.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5977999999999998e-05 2023-10-20T03:36:17.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.035026+0000", 2023-10-20T03:36:17.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011898633 2023-10-20T03:36:17.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.035107+0000", 2023-10-20T03:36:17.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0476000000000007e-05 2023-10-20T03:36:17.238 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.238 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.238 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.239 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s3 71/56 ECSubWrite(tid=63, reqid=client.4570.0:575, at_version=71'46, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:17.239 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:52.983574+0000", 2023-10-20T03:36:17.239 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 24.086515051999999, 2023-10-20T03:36:17.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011610976, 2023-10-20T03:36:17.240 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.240 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.240 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.241 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.983574+0000", 2023-10-20T03:36:17.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.241 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.983574+0000", 2023-10-20T03:36:17.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.983575+0000", 2023-10-20T03:36:17.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1429999999999999e-06 2023-10-20T03:36:17.243 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.244 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984011+0000", 2023-10-20T03:36:17.244 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00043591300000000001 2023-10-20T03:36:17.244 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.244 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.244 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.245 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984039+0000", 2023-10-20T03:36:17.245 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8119999999999998e-05 2023-10-20T03:36:17.245 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.245 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.246 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.246 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984052+0000", 2023-10-20T03:36:17.246 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3332e-05 2023-10-20T03:36:17.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.246 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.247 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.247 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984151+0000", 2023-10-20T03:36:17.247 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8503000000000002e-05 2023-10-20T03:36:17.247 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.247 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.248 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.248 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984168+0000", 2023-10-20T03:36:17.248 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7053000000000001e-05 2023-10-20T03:36:17.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.248 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.249 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.249 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.995118+0000", 2023-10-20T03:36:17.249 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010950616 2023-10-20T03:36:17.249 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.249 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.250 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.250 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.995185+0000", 2023-10-20T03:36:17.250 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6296000000000004e-05 2023-10-20T03:36:17.250 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.250 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.251 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.251 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.251 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s2 74/31 ECSubWrite(tid=65, reqid=client.4570.0:615, at_version=74'48, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:17.251 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:56.030516+0000", 2023-10-20T03:36:17.252 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.039572282000002, 2023-10-20T03:36:17.252 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010678188, 2023-10-20T03:36:17.252 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.252 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.252 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.253 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.253 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.253 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030516+0000", 2023-10-20T03:36:17.253 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.254 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.254 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.254 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030516+0000", 2023-10-20T03:36:17.254 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.254 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.255 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.255 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.255 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030517+0000", 2023-10-20T03:36:17.255 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.4799999999999997e-07 2023-10-20T03:36:17.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.256 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.256 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030941+0000", 2023-10-20T03:36:17.256 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00042401400000000001 2023-10-20T03:36:17.257 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.257 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.257 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.257 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030956+0000", 2023-10-20T03:36:17.257 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4381e-05 2023-10-20T03:36:17.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.258 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.258 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.258 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030963+0000", 2023-10-20T03:36:17.258 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3499999999999999e-06 2023-10-20T03:36:17.259 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.259 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.259 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.259 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.031035+0000", 2023-10-20T03:36:17.259 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1512000000000002e-05 2023-10-20T03:36:17.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.260 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.260 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.260 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.031048+0000", 2023-10-20T03:36:17.261 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3118e-05 2023-10-20T03:36:17.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.261 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.261 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.261 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.041148+0000", 2023-10-20T03:36:17.262 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010100029 2023-10-20T03:36:17.262 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.262 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.262 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.262 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.041195+0000", 2023-10-20T03:36:17.262 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6935999999999999e-05 2023-10-20T03:36:17.263 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.263 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.263 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.263 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.263 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.264 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s2 91/31 ECSubWrite(tid=93, reqid=client.4570.0:781, at_version=91'71, trim_to=0'0, roll_forward_to=88'69))", 2023-10-20T03:36:17.264 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:13.134755+0000", 2023-10-20T03:36:17.264 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.9353334800000002, 2023-10-20T03:36:17.264 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010955094, 2023-10-20T03:36:17.264 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.265 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.265 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.265 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.265 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.266 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.134755+0000", 2023-10-20T03:36:17.266 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.266 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.266 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.267 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.134755+0000", 2023-10-20T03:36:17.267 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.267 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.267 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.267 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.268 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.134756+0000", 2023-10-20T03:36:17.268 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.8400000000000004e-07 2023-10-20T03:36:17.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.268 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.268 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.269 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.134811+0000", 2023-10-20T03:36:17.269 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5081000000000003e-05 2023-10-20T03:36:17.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.269 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.269 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.270 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.134845+0000", 2023-10-20T03:36:17.270 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4449000000000002e-05 2023-10-20T03:36:17.270 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.270 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.270 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.271 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.134867+0000", 2023-10-20T03:36:17.271 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1546999999999999e-05 2023-10-20T03:36:17.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.271 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.271 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.272 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.135034+0000", 2023-10-20T03:36:17.272 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016751099999999999 2023-10-20T03:36:17.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.272 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.272 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.272 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.135068+0000", 2023-10-20T03:36:17.273 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3835999999999999e-05 2023-10-20T03:36:17.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.273 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.273 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.273 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.145612+0000", 2023-10-20T03:36:17.274 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010543861999999999 2023-10-20T03:36:17.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.274 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.274 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.274 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.145710+0000", 2023-10-20T03:36:17.275 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.8123999999999998e-05 2023-10-20T03:36:17.275 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.275 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.275 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.276 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.276 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s2 91/31 ECSubWrite(tid=94, reqid=client.4570.0:782, at_version=91'72, trim_to=0'0, roll_forward_to=88'69))", 2023-10-20T03:36:17.276 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:13.140031+0000", 2023-10-20T03:36:17.276 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 3.930057862, 2023-10-20T03:36:17.277 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099742310000000001, 2023-10-20T03:36:17.277 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.277 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.277 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.277 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.278 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.278 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.140031+0000", 2023-10-20T03:36:17.278 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.278 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.279 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.279 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.140031+0000", 2023-10-20T03:36:17.279 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.279 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.280 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.280 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.140032+0000", 2023-10-20T03:36:17.280 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5e-06 2023-10-20T03:36:17.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.280 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.281 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.281 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.140182+0000", 2023-10-20T03:36:17.281 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014940600000000001 2023-10-20T03:36:17.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.281 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.282 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.282 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.140216+0000", 2023-10-20T03:36:17.282 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4365e-05 2023-10-20T03:36:17.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.282 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.283 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.283 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.140239+0000", 2023-10-20T03:36:17.283 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.27e-05 2023-10-20T03:36:17.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.283 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.284 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.284 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.140496+0000", 2023-10-20T03:36:17.284 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000257175 2023-10-20T03:36:17.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.284 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.285 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.285 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.140545+0000", 2023-10-20T03:36:17.285 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8992000000000001e-05 2023-10-20T03:36:17.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.285 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.286 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.286 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.149895+0000", 2023-10-20T03:36:17.286 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093495690000000003 2023-10-20T03:36:17.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.286 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.150005+0000", 2023-10-20T03:36:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000110524 2023-10-20T03:36:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.288 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.288 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.288 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.288 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds3 92/56 ECSubWrite(tid=98, reqid=osd.1.0:98, at_version=92'69, trim_to=0'0, roll_forward_to=88'67))", 2023-10-20T03:36:17.288 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:14.150537+0000", 2023-10-20T03:36:17.289 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 2.919551491, 2023-10-20T03:36:17.289 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099847040000000005, 2023-10-20T03:36:17.289 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.289 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.289 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.290 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.290 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.290 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.150537+0000", 2023-10-20T03:36:17.290 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.291 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.291 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.291 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.150537+0000", 2023-10-20T03:36:17.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.292 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.292 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.150538+0000", 2023-10-20T03:36:17.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4700000000000001e-07 2023-10-20T03:36:17.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.150592+0000", 2023-10-20T03:36:17.293 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4029000000000001e-05 2023-10-20T03:36:17.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.294 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.294 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.294 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.150633+0000", 2023-10-20T03:36:17.294 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1004999999999998e-05 2023-10-20T03:36:17.294 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.295 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.295 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.295 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.150659+0000", 2023-10-20T03:36:17.295 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5908999999999999e-05 2023-10-20T03:36:17.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.296 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.296 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.150871+0000", 2023-10-20T03:36:17.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00021184700000000001 2023-10-20T03:36:17.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.150910+0000", 2023-10-20T03:36:17.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9604999999999998e-05 2023-10-20T03:36:17.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.160394+0000", 2023-10-20T03:36:17.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0094834189999999999 2023-10-20T03:36:17.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.160522+0000", 2023-10-20T03:36:17.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000128143 2023-10-20T03:36:17.300 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.300 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.300 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.300 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s2 92/31 ECSubWrite(tid=97, reqid=osd.14.0:97, at_version=92'75, trim_to=0'0, roll_forward_to=91'73))", 2023-10-20T03:36:17.301 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:14.151191+0000", 2023-10-20T03:36:17.301 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 2.9188972509999997, 2023-10-20T03:36:17.301 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0093522129999999998, 2023-10-20T03:36:17.301 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.302 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.302 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.302 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.302 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151191+0000", 2023-10-20T03:36:17.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.303 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151191+0000", 2023-10-20T03:36:17.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151192+0000", 2023-10-20T03:36:17.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.3700000000000001e-07 2023-10-20T03:36:17.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.306 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151221+0000", 2023-10-20T03:36:17.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9493e-05 2023-10-20T03:36:17.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.306 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.307 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151240+0000", 2023-10-20T03:36:17.307 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8660000000000001e-05 2023-10-20T03:36:17.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.307 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.307 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151253+0000", 2023-10-20T03:36:17.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3392e-05 2023-10-20T03:36:17.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151407+0000", 2023-10-20T03:36:17.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015380999999999999 2023-10-20T03:36:17.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151426+0000", 2023-10-20T03:36:17.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8743999999999999e-05 2023-10-20T03:36:17.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.160395+0000", 2023-10-20T03:36:17.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.008969065 2023-10-20T03:36:17.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.160544+0000", 2023-10-20T03:36:17.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014861200000000001 2023-10-20T03:36:17.312 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.312 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.312 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.313 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s3 92/56 ECSubWrite(tid=99, reqid=osd.10.0:99, at_version=92'71, trim_to=0'0, roll_forward_to=88'69))", 2023-10-20T03:36:17.313 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:14.152866+0000", 2023-10-20T03:36:17.313 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 2.9172224820000001, 2023-10-20T03:36:17.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0098777680000000003, 2023-10-20T03:36:17.314 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.314 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.314 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.152866+0000", 2023-10-20T03:36:17.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.152866+0000", 2023-10-20T03:36:17.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.317 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.152866+0000", 2023-10-20T03:36:17.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5800000000000001e-07 2023-10-20T03:36:17.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.152894+0000", 2023-10-20T03:36:17.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7881999999999998e-05 2023-10-20T03:36:17.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.319 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.319 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.152913+0000", 2023-10-20T03:36:17.319 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8410000000000002e-05 2023-10-20T03:36:17.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.320 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.152926+0000", 2023-10-20T03:36:17.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2904999999999999e-05 2023-10-20T03:36:17.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.153375+0000", 2023-10-20T03:36:17.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00044981900000000001 2023-10-20T03:36:17.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.322 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.153408+0000", 2023-10-20T03:36:17.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2988999999999999e-05 2023-10-20T03:36:17.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.323 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.323 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.162686+0000", 2023-10-20T03:36:17.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0092771729999999997 2023-10-20T03:36:17.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.324 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.324 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.324 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.162744+0000", 2023-10-20T03:36:17.324 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8332000000000003e-05 2023-10-20T03:36:17.324 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.325 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.325 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.325 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.325 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.326 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:17.326 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:36:17.327 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:36:17.328 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-10-20T03:36:17.328 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:36:17.329 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:36:17.329 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:36:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:36:17.330 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:36:17.331 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:36:17.331 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-10-20T03:36:17.332 INFO:tasks.rados.rados.0.smithi170.stdout:334: done (0 left) 2023-10-20T03:36:17.333 INFO:tasks.rados.rados.0.smithi170.stdout:335: read oid 1 snap -1 2023-10-20T03:36:17.333 INFO:tasks.rados.rados.0.smithi170.stdout:335: expect deleted 2023-10-20T03:36:17.333 INFO:tasks.rados.rados.0.smithi170.stdout:336: delete oid 18 current snap is 31 2023-10-20T03:36:17.333 INFO:tasks.rados.rados.0.smithi170.stdout:335: done (1 left) 2023-10-20T03:36:17.333 INFO:tasks.rados.rados.0.smithi170.stdout:336: done (0 left) 2023-10-20T03:36:17.334 INFO:tasks.rados.rados.0.smithi170.stdout:337: setattr oid 8 current snap is 31 2023-10-20T03:36:17.334 INFO:tasks.rados.rados.0.smithi170.stdout:338: snap_remove snap 27 2023-10-20T03:36:17.334 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:17.331+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:17.432 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-10-20T03:36:17.511 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:17.511 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:36:17.512 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:36:17.512 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:17.530 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-10-20T03:36:17.535 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:17.528+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:17.638 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:17.636+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:17.645 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:17.645 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:36:17.646 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:36:17.646 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:36:17.646 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:17.660 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-10-20T03:36:17.739 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:17.735+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:17.822 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:17.822 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:36:17.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:36:17.823 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:36:17.823 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.823 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds1 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:36:17.823 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.392154+0000", 2023-10-20T03:36:17.824 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 47.426673786999999, 2023-10-20T03:36:17.824 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.055444475, 2023-10-20T03:36:17.824 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.824 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.824 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.825 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.825 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392154+0000", 2023-10-20T03:36:17.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.826 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.826 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392154+0000", 2023-10-20T03:36:17.826 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.827 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.827 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.827 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392156+0000", 2023-10-20T03:36:17.827 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0499999999999999e-06 2023-10-20T03:36:17.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.828 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.828 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.828 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392978+0000", 2023-10-20T03:36:17.828 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00082210300000000003 2023-10-20T03:36:17.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.829 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393026+0000", 2023-10-20T03:36:17.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8306000000000001e-05 2023-10-20T03:36:17.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393053+0000", 2023-10-20T03:36:17.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6872e-05 2023-10-20T03:36:17.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393248+0000", 2023-10-20T03:36:17.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019507500000000001 2023-10-20T03:36:17.832 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.832 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.832 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.832 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393331+0000", 2023-10-20T03:36:17.832 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2701999999999995e-05 2023-10-20T03:36:17.833 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.833 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.833 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.447508+0000", 2023-10-20T03:36:17.833 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054176766000000001 2023-10-20T03:36:17.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.834 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.834 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.834 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.447598+0000", 2023-10-20T03:36:17.834 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0600999999999995e-05 2023-10-20T03:36:17.835 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.835 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.835 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.836 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 49/31 ECSubWrite(tid=39, reqid=client.4570.0:390, at_version=49'23, trim_to=0'0, roll_forward_to=47'21))", 2023-10-20T03:36:17.836 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.413323+0000", 2023-10-20T03:36:17.836 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 46.405504665000002, 2023-10-20T03:36:17.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012748729, 2023-10-20T03:36:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.413323+0000", 2023-10-20T03:36:17.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.839 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.413323+0000", 2023-10-20T03:36:17.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.839 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.413324+0000", 2023-10-20T03:36:17.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0699999999999999e-06 2023-10-20T03:36:17.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.842 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.842 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414070+0000", 2023-10-20T03:36:17.842 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00074601599999999995 2023-10-20T03:36:17.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.843 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.843 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414089+0000", 2023-10-20T03:36:17.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9199999999999999e-05 2023-10-20T03:36:17.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.844 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414102+0000", 2023-10-20T03:36:17.844 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2099000000000001e-05 2023-10-20T03:36:17.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.845 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.845 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414201+0000", 2023-10-20T03:36:17.845 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.9931000000000003e-05 2023-10-20T03:36:17.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.414214+0000", 2023-10-20T03:36:17.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2652e-05 2023-10-20T03:36:17.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.426026+0000", 2023-10-20T03:36:17.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011811845 2023-10-20T03:36:17.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.426072+0000", 2023-10-20T03:36:17.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5915999999999999e-05 2023-10-20T03:36:17.848 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.849 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.849 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.849 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 54/31 ECSubWrite(tid=65, reqid=client.4570.0:418, at_version=54'32, trim_to=0'0, roll_forward_to=51'31))", 2023-10-20T03:36:17.850 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:36.458762+0000", 2023-10-20T03:36:17.850 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.360065908000003, 2023-10-20T03:36:17.850 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015096593, 2023-10-20T03:36:17.850 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.850 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.851 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.458762+0000", 2023-10-20T03:36:17.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.458762+0000", 2023-10-20T03:36:17.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.853 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.853 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.458764+0000", 2023-10-20T03:36:17.853 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6700000000000001e-06 2023-10-20T03:36:17.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.854 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.854 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.854 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.459218+0000", 2023-10-20T03:36:17.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00045411099999999999 2023-10-20T03:36:17.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.855 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.855 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.459248+0000", 2023-10-20T03:36:17.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0828999999999999e-05 2023-10-20T03:36:17.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.856 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.856 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.459264+0000", 2023-10-20T03:36:17.856 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5143000000000001e-05 2023-10-20T03:36:17.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.459379+0000", 2023-10-20T03:36:17.857 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00011584800000000001 2023-10-20T03:36:17.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.459398+0000", 2023-10-20T03:36:17.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8185999999999998e-05 2023-10-20T03:36:17.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.859 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.859 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.473773+0000", 2023-10-20T03:36:17.859 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014375221000000001 2023-10-20T03:36:17.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:36.473858+0000", 2023-10-20T03:36:17.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5585000000000001e-05 2023-10-20T03:36:17.861 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.861 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.861 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.861 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.bs5 60/56 ECSubWrite(tid=208, reqid=client.4570.0:471, at_version=60'25, trim_to=0'0, roll_forward_to=60'22))", 2023-10-20T03:36:17.862 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.933491+0000", 2023-10-20T03:36:17.862 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.885336918999997, 2023-10-20T03:36:17.862 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015929024999999999, 2023-10-20T03:36:17.862 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.862 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.863 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.863 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.863 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.863 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933491+0000", 2023-10-20T03:36:17.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.864 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.864 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.864 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933491+0000", 2023-10-20T03:36:17.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933492+0000", 2023-10-20T03:36:17.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.331e-06 2023-10-20T03:36:17.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933641+0000", 2023-10-20T03:36:17.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014902599999999999 2023-10-20T03:36:17.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933669+0000", 2023-10-20T03:36:17.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8107999999999999e-05 2023-10-20T03:36:17.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933685+0000", 2023-10-20T03:36:17.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5319999999999999e-05 2023-10-20T03:36:17.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933835+0000", 2023-10-20T03:36:17.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015027700000000001 2023-10-20T03:36:17.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.933852+0000", 2023-10-20T03:36:17.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6965000000000001e-05 2023-10-20T03:36:17.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.871 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.871 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.949309+0000", 2023-10-20T03:36:17.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015457256000000001 2023-10-20T03:36:17.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.872 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.872 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.949420+0000", 2023-10-20T03:36:17.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000110742 2023-10-20T03:36:17.872 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.872 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.873 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.873 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.bs5 60/56 ECSubWrite(tid=211, reqid=client.4570.0:472, at_version=60'26, trim_to=0'0, roll_forward_to=60'22))", 2023-10-20T03:36:17.873 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:41.939803+0000", 2023-10-20T03:36:17.874 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 35.879025202999998, 2023-10-20T03:36:17.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013356855000000001, 2023-10-20T03:36:17.874 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.874 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.874 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.939803+0000", 2023-10-20T03:36:17.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.939803+0000", 2023-10-20T03:36:17.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.939805+0000", 2023-10-20T03:36:17.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1569999999999998e-06 2023-10-20T03:36:17.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.940088+0000", 2023-10-20T03:36:17.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028281699999999999 2023-10-20T03:36:17.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.940123+0000", 2023-10-20T03:36:17.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5930999999999999e-05 2023-10-20T03:36:17.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.880 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.880 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.940145+0000", 2023-10-20T03:36:17.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1299999999999999e-05 2023-10-20T03:36:17.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.880 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.880 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.881 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.941352+0000", 2023-10-20T03:36:17.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001207381 2023-10-20T03:36:17.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.882 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.882 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.941370+0000", 2023-10-20T03:36:17.882 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7666999999999999e-05 2023-10-20T03:36:17.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.953053+0000", 2023-10-20T03:36:17.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011683321 2023-10-20T03:36:17.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:41.953159+0000", 2023-10-20T03:36:17.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010628099999999999 2023-10-20T03:36:17.884 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.884 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.884 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.885 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 69/31 ECSubWrite(tid=95, reqid=client.4570.0:552, at_version=69'41, trim_to=0'0, roll_forward_to=56'39))", 2023-10-20T03:36:17.885 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:50.953532+0000", 2023-10-20T03:36:17.885 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 26.865295388, 2023-10-20T03:36:17.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017017713, 2023-10-20T03:36:17.886 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.886 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.886 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.953532+0000", 2023-10-20T03:36:17.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.953532+0000", 2023-10-20T03:36:17.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.953533+0000", 2023-10-20T03:36:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1999999999999999e-07 2023-10-20T03:36:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.889 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.890 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.890 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.954582+0000", 2023-10-20T03:36:17.890 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001049094 2023-10-20T03:36:17.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.891 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.891 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.954597+0000", 2023-10-20T03:36:17.891 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5147000000000001e-05 2023-10-20T03:36:17.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.892 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.954606+0000", 2023-10-20T03:36:17.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9570000000000008e-06 2023-10-20T03:36:17.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.893 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.954681+0000", 2023-10-20T03:36:17.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4901000000000007e-05 2023-10-20T03:36:17.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.954689+0000", 2023-10-20T03:36:17.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7870000000000007e-06 2023-10-20T03:36:17.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.970508+0000", 2023-10-20T03:36:17.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015818249999999999 2023-10-20T03:36:17.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:50.970550+0000", 2023-10-20T03:36:17.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2456999999999997e-05 2023-10-20T03:36:17.896 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.896 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.896 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.897 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.897 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s3 74/31 ECSubWrite(tid=65, reqid=client.4570.0:615, at_version=74'48, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:17.897 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:56.030770+0000", 2023-10-20T03:36:17.897 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 21.788057772999998, 2023-10-20T03:36:17.898 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.021243306999999999, 2023-10-20T03:36:17.898 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.898 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.898 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.898 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.899 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.899 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030770+0000", 2023-10-20T03:36:17.899 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.899 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.900 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030770+0000", 2023-10-20T03:36:17.900 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.900 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.901 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.030771+0000", 2023-10-20T03:36:17.901 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0800000000000003e-07 2023-10-20T03:36:17.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.901 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.901 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.902 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.031306+0000", 2023-10-20T03:36:17.902 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053512700000000004 2023-10-20T03:36:17.902 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.902 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.902 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.903 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.031324+0000", 2023-10-20T03:36:17.903 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8338999999999998e-05 2023-10-20T03:36:17.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.903 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.903 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.903 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.031335+0000", 2023-10-20T03:36:17.904 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0764999999999999e-05 2023-10-20T03:36:17.904 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.904 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.904 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.904 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.031504+0000", 2023-10-20T03:36:17.905 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016858800000000001 2023-10-20T03:36:17.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.905 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.905 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.905 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.031533+0000", 2023-10-20T03:36:17.906 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8789000000000002e-05 2023-10-20T03:36:17.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.906 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.906 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.906 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.051981+0000", 2023-10-20T03:36:17.907 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020448663999999998 2023-10-20T03:36:17.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.907 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.907 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:56.052013+0000", 2023-10-20T03:36:17.907 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2126999999999999e-05 2023-10-20T03:36:17.908 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.908 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.908 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.908 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.908 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.909 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.0s2 75/31 ECSubWrite(tid=97, reqid=client.4570.0:628, at_version=75'43, trim_to=0'0, roll_forward_to=69'41))", 2023-10-20T03:36:17.909 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:57.035627+0000", 2023-10-20T03:36:17.909 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.783201085999998, 2023-10-20T03:36:17.909 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018530971, 2023-10-20T03:36:17.909 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.910 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.910 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.910 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.910 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.910 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.035627+0000", 2023-10-20T03:36:17.911 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.911 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.911 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.035627+0000", 2023-10-20T03:36:17.912 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.912 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.912 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.035628+0000", 2023-10-20T03:36:17.912 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4270000000000001e-06 2023-10-20T03:36:17.913 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.913 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.913 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.913 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.037173+0000", 2023-10-20T03:36:17.913 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0015447620000000001 2023-10-20T03:36:17.913 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.914 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.914 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.914 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.037202+0000", 2023-10-20T03:36:17.914 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9246000000000001e-05 2023-10-20T03:36:17.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.915 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.915 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.915 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.037218+0000", 2023-10-20T03:36:17.915 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5359999999999999e-05 2023-10-20T03:36:17.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.916 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.916 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.916 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.037321+0000", 2023-10-20T03:36:17.916 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010364199999999999 2023-10-20T03:36:17.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.917 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.917 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:17.917 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.037334+0000", 2023-10-20T03:36:17.917 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2595000000000001e-05 2023-10-20T03:36:17.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.918 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:17.918 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.054118+0000", 2023-10-20T03:36:17.918 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016784054 2023-10-20T03:36:17.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.919 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.919 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.919 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.054158+0000", 2023-10-20T03:36:17.919 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9885000000000002e-05 2023-10-20T03:36:17.919 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.919 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.920 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.920 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.920 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDPGPush(3.6s4 75/74 [PushOp(3:6d58622e:::smithi17039300-33:b, version: 44'28, data_included: [0~221184], data_size: 221184, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:6d58622e:::smithi17039300-33:b@44'28, size: 884736, copy_subset: [], clone_subset: {}, snapset: 19=[]:{b: [b, a, 9, 6, 3, 2]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 884736, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-10-20T03:36:17.920 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:57.092260+0000", 2023-10-20T03:36:17.921 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.726568062999998, 2023-10-20T03:36:17.921 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015275433999999999, 2023-10-20T03:36:17.921 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.921 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.921 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.921 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.922 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.922 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.092260+0000", 2023-10-20T03:36:17.922 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.923 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.923 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.923 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.092260+0000", 2023-10-20T03:36:17.923 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.924 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.924 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.924 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.092261+0000", 2023-10-20T03:36:17.924 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1900000000000001e-07 2023-10-20T03:36:17.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.925 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.925 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.925 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.092451+0000", 2023-10-20T03:36:17.925 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00019055400000000001 2023-10-20T03:36:17.925 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.926 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.926 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.926 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.092478+0000", 2023-10-20T03:36:17.926 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7131000000000002e-05 2023-10-20T03:36:17.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.927 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.927 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.927 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.092496+0000", 2023-10-20T03:36:17.927 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7331000000000001e-05 2023-10-20T03:36:17.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.928 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.928 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.928 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.092661+0000", 2023-10-20T03:36:17.928 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000165279 2023-10-20T03:36:17.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.929 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.929 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.929 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.107535+0000", 2023-10-20T03:36:17.929 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01487422 2023-10-20T03:36:17.929 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.930 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.930 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.930 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.930 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.930 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDPGPush(3.6s4 75/74 [PushOp(3:62cf25fd:::smithi17039300-14:c, version: 45'37, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:62cf25fd:::smithi17039300-14:c@45'37, size: 1572864, copy_subset: [], clone_subset: {}, snapset: 19=[]:{6: [6, 3, 2], a: [a, 9], c: [c, b]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-10-20T03:36:17.931 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:57.159472+0000", 2023-10-20T03:36:17.931 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.659355390999998, 2023-10-20T03:36:17.931 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017948883999999998, 2023-10-20T03:36:17.931 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.931 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.932 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.932 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.932 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.932 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.159472+0000", 2023-10-20T03:36:17.932 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.933 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.933 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.933 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.933 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.159472+0000", 2023-10-20T03:36:17.933 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.934 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.934 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.934 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.934 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.159474+0000", 2023-10-20T03:36:17.934 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.57e-06 2023-10-20T03:36:17.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.935 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.935 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.935 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.159660+0000", 2023-10-20T03:36:17.935 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018636999999999999 2023-10-20T03:36:17.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.936 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.936 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.936 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.159709+0000", 2023-10-20T03:36:17.936 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8423e-05 2023-10-20T03:36:17.937 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.937 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.937 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.937 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.159738+0000", 2023-10-20T03:36:17.937 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9519e-05 2023-10-20T03:36:17.937 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.938 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.938 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.938 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.159972+0000", 2023-10-20T03:36:17.938 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000233284 2023-10-20T03:36:17.938 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.939 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.939 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.939 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.177421+0000", 2023-10-20T03:36:17.939 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017449717999999999 2023-10-20T03:36:17.939 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.940 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.940 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.940 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.940 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDPGPush(3.6s4 75/74 [PushOp(3:62cf25fd:::smithi17039300-14:head, version: 61'47, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:62cf25fd:::smithi17039300-14:head@61'47, size: 4702208, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-10-20T03:36:17.941 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:57.222267+0000", 2023-10-20T03:36:17.941 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.5965612, 2023-10-20T03:36:17.941 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018035045999999999, 2023-10-20T03:36:17.941 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.941 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.942 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.942 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.942 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.942 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.222267+0000", 2023-10-20T03:36:17.942 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.943 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.943 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.943 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.222267+0000", 2023-10-20T03:36:17.943 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.944 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.944 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.944 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.944 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.222268+0000", 2023-10-20T03:36:17.944 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.703e-06 2023-10-20T03:36:17.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.945 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.945 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.945 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.222665+0000", 2023-10-20T03:36:17.945 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00039635499999999998 2023-10-20T03:36:17.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.946 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.946 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.946 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.222707+0000", 2023-10-20T03:36:17.946 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.1919000000000003e-05 2023-10-20T03:36:17.946 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.947 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.947 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.947 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.222734+0000", 2023-10-20T03:36:17.947 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7460000000000001e-05 2023-10-20T03:36:17.947 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.948 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.948 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.948 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.222920+0000", 2023-10-20T03:36:17.948 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018606699999999999 2023-10-20T03:36:17.948 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.949 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.949 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.949 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.240302+0000", 2023-10-20T03:36:17.949 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017381542 2023-10-20T03:36:17.949 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.949 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.950 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.950 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.950 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.950 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDPGPush(3.6s4 75/74 [PushOp(3:62cf25fd:::smithi17039300-14:head, version: 61'47, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:62cf25fd:::smithi17039300-14:head@61'47, size: 4702208, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-10-20T03:36:17.950 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:57.245296+0000", 2023-10-20T03:36:17.951 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.573531312, 2023-10-20T03:36:17.951 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018534609000000001, 2023-10-20T03:36:17.951 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.951 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.951 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.952 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.952 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.952 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.245296+0000", 2023-10-20T03:36:17.952 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.952 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.953 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.953 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.953 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.245296+0000", 2023-10-20T03:36:17.953 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.954 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.954 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.954 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.954 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.245298+0000", 2023-10-20T03:36:17.954 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6840000000000001e-06 2023-10-20T03:36:17.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.955 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.955 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.955 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.245671+0000", 2023-10-20T03:36:17.955 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00037270800000000002 2023-10-20T03:36:17.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.956 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.956 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.956 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.245718+0000", 2023-10-20T03:36:17.956 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7169000000000002e-05 2023-10-20T03:36:17.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.957 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.957 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.957 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.245745+0000", 2023-10-20T03:36:17.957 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7206999999999999e-05 2023-10-20T03:36:17.957 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.958 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.958 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.958 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.245923+0000", 2023-10-20T03:36:17.958 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00017780300000000001 2023-10-20T03:36:17.958 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.959 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.959 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.959 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.263831+0000", 2023-10-20T03:36:17.959 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017908038000000001 2023-10-20T03:36:17.959 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.959 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.960 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.960 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.960 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDPGPush(3.6s4 75/74 [PushOp(3:62cf25fd:::smithi17039300-14:head, version: 61'47, data_included: [524288~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:62cf25fd:::smithi17039300-14:head@61'47, size: 4702208, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-10-20T03:36:17.960 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:57.269018+0000", 2023-10-20T03:36:17.961 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.549809958000001, 2023-10-20T03:36:17.961 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017761956999999998, 2023-10-20T03:36:17.961 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.961 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.961 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.962 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.962 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.962 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.269018+0000", 2023-10-20T03:36:17.962 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.963 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.963 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.963 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.269018+0000", 2023-10-20T03:36:17.963 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.963 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.963 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.964 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.964 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.269019+0000", 2023-10-20T03:36:17.964 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.457e-06 2023-10-20T03:36:17.964 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.964 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.965 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.965 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.269401+0000", 2023-10-20T03:36:17.965 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00038191199999999998 2023-10-20T03:36:17.965 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.965 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.966 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.966 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.269448+0000", 2023-10-20T03:36:17.966 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6702000000000001e-05 2023-10-20T03:36:17.966 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.966 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.966 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.967 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.269479+0000", 2023-10-20T03:36:17.967 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0902999999999999e-05 2023-10-20T03:36:17.967 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.967 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.967 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.968 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.269661+0000", 2023-10-20T03:36:17.968 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018239800000000001 2023-10-20T03:36:17.968 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.968 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.968 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.969 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.286780+0000", 2023-10-20T03:36:17.969 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017118584999999999 2023-10-20T03:36:17.969 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.969 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.969 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.970 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.970 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.970 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDPGPush(3.6s4 75/74 [PushOp(3:62cf25fd:::smithi17039300-14:head, version: 61'47, data_included: [786432~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:62cf25fd:::smithi17039300-14:head@61'47, size: 4702208, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-10-20T03:36:17.970 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:57.291908+0000", 2023-10-20T03:36:17.970 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.526920234999999, 2023-10-20T03:36:17.970 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017998453000000001, 2023-10-20T03:36:17.971 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.971 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.971 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.971 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.971 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.972 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.291908+0000", 2023-10-20T03:36:17.972 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.972 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.972 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.972 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.972 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.291908+0000", 2023-10-20T03:36:17.973 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.973 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.973 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.973 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.973 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.291909+0000", 2023-10-20T03:36:17.974 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4699999999999999e-06 2023-10-20T03:36:17.974 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.974 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.974 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.974 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.292335+0000", 2023-10-20T03:36:17.975 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00042567100000000002 2023-10-20T03:36:17.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.975 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.975 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.975 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.292381+0000", 2023-10-20T03:36:17.976 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6628999999999997e-05 2023-10-20T03:36:17.976 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.976 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.976 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.976 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.292409+0000", 2023-10-20T03:36:17.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8005999999999999e-05 2023-10-20T03:36:17.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.977 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.977 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.977 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.292591+0000", 2023-10-20T03:36:17.977 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018178199999999999 2023-10-20T03:36:17.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.978 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.978 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.978 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.309906+0000", 2023-10-20T03:36:17.978 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017314895 2023-10-20T03:36:17.979 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.979 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.979 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.979 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.980 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDPGPush(3.6s4 75/74 [PushOp(3:6d58622e:::smithi17039300-33:head, version: 61'52, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 24, recovery_info: ObjectRecoveryInfo(3:6d58622e:::smithi17039300-33:head@61'52, size: 2588672, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-10-20T03:36:17.980 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:57.330530+0000", 2023-10-20T03:36:17.980 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.488297561, 2023-10-20T03:36:17.980 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018214580000000001, 2023-10-20T03:36:17.980 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.981 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.981 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.981 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.981 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.981 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.330530+0000", 2023-10-20T03:36:17.981 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.982 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.982 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.982 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.330530+0000", 2023-10-20T03:36:17.982 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.983 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.983 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.983 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.330532+0000", 2023-10-20T03:36:17.983 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5600000000000001e-06 2023-10-20T03:36:17.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.984 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.984 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.984 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.331032+0000", 2023-10-20T03:36:17.984 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00050036000000000004 2023-10-20T03:36:17.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.985 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.985 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.985 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.331079+0000", 2023-10-20T03:36:17.985 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7296000000000003e-05 2023-10-20T03:36:17.985 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.986 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.986 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.986 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.331107+0000", 2023-10-20T03:36:17.986 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7399e-05 2023-10-20T03:36:17.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.987 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.987 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.987 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.331296+0000", 2023-10-20T03:36:17.987 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018884800000000001 2023-10-20T03:36:17.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.988 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.988 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.988 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.348745+0000", 2023-10-20T03:36:17.988 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017449117 2023-10-20T03:36:17.988 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.989 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.989 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.989 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.989 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDPGPush(3.6s4 75/74 [PushOp(3:6d58622e:::smithi17039300-33:head, version: 61'52, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:6d58622e:::smithi17039300-33:head@61'52, size: 2588672, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-10-20T03:36:17.989 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:57.354309+0000", 2023-10-20T03:36:17.990 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.464518907999999, 2023-10-20T03:36:17.990 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017800223, 2023-10-20T03:36:17.990 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:17.990 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:17.990 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:17.991 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.991 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:17.991 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.354309+0000", 2023-10-20T03:36:17.991 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.991 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.992 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.992 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:17.992 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.354309+0000", 2023-10-20T03:36:17.992 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:17.992 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.993 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.993 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:17.993 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.354310+0000", 2023-10-20T03:36:17.993 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5600000000000001e-06 2023-10-20T03:36:17.993 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.993 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.994 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:17.994 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.354701+0000", 2023-10-20T03:36:17.994 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00039064500000000001 2023-10-20T03:36:17.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.995 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.995 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:17.995 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.354748+0000", 2023-10-20T03:36:17.995 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6632999999999999e-05 2023-10-20T03:36:17.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.995 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.996 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:17.996 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.354776+0000", 2023-10-20T03:36:17.996 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8036000000000001e-05 2023-10-20T03:36:17.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.996 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.997 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:17.997 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.354960+0000", 2023-10-20T03:36:17.997 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018462099999999999 2023-10-20T03:36:17.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.997 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.997 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:17.998 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.372109+0000", 2023-10-20T03:36:17.998 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017148727999999999 2023-10-20T03:36:17.998 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.998 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:17.998 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:17.999 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:17.999 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:17.999 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDPGPush(3.6s4 75/74 [PushOp(3:64df2e41:::smithi17039300-17:head, version: 72'57, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 18, recovery_info: ObjectRecoveryInfo(3:64df2e41:::smithi17039300-17:head@72'57, size: 2195456, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-10-20T03:36:17.999 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:57.408675+0000", 2023-10-20T03:36:17.999 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.410152846999999, 2023-10-20T03:36:18.000 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018178248000000001, 2023-10-20T03:36:18.000 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.000 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.000 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.000 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.001 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.001 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.408675+0000", 2023-10-20T03:36:18.001 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.001 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.002 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.002 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.408675+0000", 2023-10-20T03:36:18.002 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.002 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.002 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.003 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.003 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.408676+0000", 2023-10-20T03:36:18.003 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.533e-06 2023-10-20T03:36:18.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.004 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.004 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.004 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.409089+0000", 2023-10-20T03:36:18.004 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00041290199999999998 2023-10-20T03:36:18.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.005 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.005 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.005 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.409136+0000", 2023-10-20T03:36:18.005 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6232000000000001e-05 2023-10-20T03:36:18.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.006 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.006 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.006 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.409163+0000", 2023-10-20T03:36:18.006 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6950000000000001e-05 2023-10-20T03:36:18.006 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.007 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.007 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.007 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.409338+0000", 2023-10-20T03:36:18.007 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000175551 2023-10-20T03:36:18.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.008 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.008 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.008 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.426853+0000", 2023-10-20T03:36:18.008 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017515079999999999 2023-10-20T03:36:18.008 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.008 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.009 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.009 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.009 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.009 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDPGPush(3.6s4 75/74 [PushOp(3:64df2e41:::smithi17039300-17:head, version: 72'57, data_included: [262144~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:64df2e41:::smithi17039300-17:head@72'57, size: 2195456, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-10-20T03:36:18.009 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:57.431904+0000", 2023-10-20T03:36:18.010 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 20.386924049000001, 2023-10-20T03:36:18.010 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017706013999999999, 2023-10-20T03:36:18.010 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.010 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.010 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.011 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.011 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.011 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.431904+0000", 2023-10-20T03:36:18.011 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.011 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.012 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.012 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.431904+0000", 2023-10-20T03:36:18.012 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.012 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.012 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.013 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.013 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.431905+0000", 2023-10-20T03:36:18.013 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5799999999999999e-06 2023-10-20T03:36:18.013 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.013 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.014 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.014 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.432289+0000", 2023-10-20T03:36:18.014 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00038363500000000001 2023-10-20T03:36:18.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.014 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.015 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.015 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.432336+0000", 2023-10-20T03:36:18.015 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6718999999999998e-05 2023-10-20T03:36:18.015 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.015 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.015 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.016 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.432363+0000", 2023-10-20T03:36:18.016 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7472999999999999e-05 2023-10-20T03:36:18.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.016 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.016 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.017 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.432532+0000", 2023-10-20T03:36:18.017 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000168448 2023-10-20T03:36:18.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.017 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.017 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.018 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:57.449610+0000", 2023-10-20T03:36:18.018 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017078158999999999 2023-10-20T03:36:18.018 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.018 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.018 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.019 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.019 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:819 3.as1 3:51752c29:::smithi17039300-1:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2023-10-20T03:36:18.019 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:17.177236+0000", 2023-10-20T03:36:18.019 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.64159217099999999, 2023-10-20T03:36:18.019 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013899824, 2023-10-20T03:36:18.020 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.020 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.020 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:18.020 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:18.020 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:18.021 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 819 2023-10-20T03:36:18.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.021 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.021 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.021 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.022 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177236+0000", 2023-10-20T03:36:18.022 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.022 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.022 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.023 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177236+0000", 2023-10-20T03:36:18.023 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.023 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.023 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.024 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177236+0000", 2023-10-20T03:36:18.024 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5799999999999999e-07 2023-10-20T03:36:18.024 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.024 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.024 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.025 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177240+0000", 2023-10-20T03:36:18.025 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3610000000000001e-06 2023-10-20T03:36:18.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.025 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.025 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.026 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177246+0000", 2023-10-20T03:36:18.026 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.6850000000000004e-06 2023-10-20T03:36:18.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.026 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.027 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.027 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177251+0000", 2023-10-20T03:36:18.027 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.668e-06 2023-10-20T03:36:18.027 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.027 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.027 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.028 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177315+0000", 2023-10-20T03:36:18.028 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3880000000000005e-05 2023-10-20T03:36:18.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.028 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.028 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2023-10-20T03:36:18.029 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177317+0000", 2023-10-20T03:36:18.029 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.782e-06 2023-10-20T03:36:18.029 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.029 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.029 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.030 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.190885+0000", 2023-10-20T03:36:18.030 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013568863 2023-10-20T03:36:18.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.030 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.030 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.031 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.191135+0000", 2023-10-20T03:36:18.031 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000250027 2023-10-20T03:36:18.031 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.031 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.031 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.032 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.032 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.032 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:820 3.as1 3:51752c29:::smithi17039300-1:head [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2023-10-20T03:36:18.032 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:17.177274+0000", 2023-10-20T03:36:18.032 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.64155420900000004, 2023-10-20T03:36:18.032 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014150866999999999, 2023-10-20T03:36:18.033 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.033 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.033 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:18.033 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:18.033 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:18.034 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 820 2023-10-20T03:36:18.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.034 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.034 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.034 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177274+0000", 2023-10-20T03:36:18.035 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.035 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.035 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.035 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.035 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177274+0000", 2023-10-20T03:36:18.036 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.036 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.036 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.036 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.036 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177274+0000", 2023-10-20T03:36:18.037 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0800000000000005e-07 2023-10-20T03:36:18.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.037 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.037 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.037 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177277+0000", 2023-10-20T03:36:18.037 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9040000000000001e-06 2023-10-20T03:36:18.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.038 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.038 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.038 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177284+0000", 2023-10-20T03:36:18.038 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.464e-06 2023-10-20T03:36:18.039 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.039 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.039 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.039 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177289+0000", 2023-10-20T03:36:18.039 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.8629999999999999e-06 2023-10-20T03:36:18.040 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.040 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.040 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.040 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177340+0000", 2023-10-20T03:36:18.040 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0767999999999999e-05 2023-10-20T03:36:18.041 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.041 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.041 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for peered", 2023-10-20T03:36:18.041 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.177341+0000", 2023-10-20T03:36:18.042 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6800000000000009e-07 2023-10-20T03:36:18.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.042 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.042 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.191203+0000", 2023-10-20T03:36:18.043 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013862375 2023-10-20T03:36:18.043 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.043 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.043 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.043 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:17.191424+0000", 2023-10-20T03:36:18.044 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000221017 2023-10-20T03:36:18.044 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.044 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.044 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.044 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.045 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.045 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:18.047 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-10-20T03:36:18.047 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:18.044+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:18.147 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:36:18.143+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:36:18.178 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2023-10-20T03:36:18.187 INFO:tasks.rados.rados.0.smithi170.stdout:338: done (1 left) 2023-10-20T03:36:18.188 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 42 current snap is 31 2023-10-20T03:36:18.188 INFO:tasks.rados.rados.0.smithi170.stdout:339: seq_num 140 ranges {3981312=802816,4784128=425984,5210112=720896,5931008=409600} 2023-10-20T03:36:18.218 INFO:tasks.rados.rados.0.smithi170.stdout:339: writing smithi17039300-42 from 3981312 to 4784128 tid 1 2023-10-20T03:36:18.221 INFO:tasks.rados.rados.0.smithi170.stdout:339: writing smithi17039300-42 from 4784128 to 5210112 tid 2 2023-10-20T03:36:18.225 INFO:tasks.rados.rados.0.smithi170.stdout:339: writing smithi17039300-42 from 5210112 to 5931008 tid 3 2023-10-20T03:36:18.227 INFO:tasks.rados.rados.0.smithi170.stdout:339: writing smithi17039300-42 from 5931008 to 6340608 tid 4 2023-10-20T03:36:18.228 INFO:tasks.rados.rados.0.smithi170.stdout:340: rollback oid 39 current snap is 31 2023-10-20T03:36:18.228 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 39 to 28 2023-10-20T03:36:18.228 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 8 v 76 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-10-20T03:36:18.228 INFO:tasks.rados.rados.0.smithi170.stdout:337: done (2 left) 2023-10-20T03:36:18.229 INFO:tasks.rados.rados.0.smithi170.stdout:341: delete oid 33 current snap is 31 2023-10-20T03:36:18.234 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:18.234 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:36:18.234 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.234 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 96/31 ECSubWrite(tid=102, reqid=client.4570.0:824, at_version=96'81, trim_to=0'0, roll_forward_to=96'79))", 2023-10-20T03:36:18.235 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:18.224278+0000", 2023-10-20T03:36:18.235 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.0093774099999999992, 2023-10-20T03:36:18.235 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0094390210000000006, 2023-10-20T03:36:18.235 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.235 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.236 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.236 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.236 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.236 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224278+0000", 2023-10-20T03:36:18.236 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.237 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.237 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.237 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224278+0000", 2023-10-20T03:36:18.237 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.238 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.238 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.238 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.238 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224279+0000", 2023-10-20T03:36:18.238 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5659999999999999e-06 2023-10-20T03:36:18.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.239 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.239 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.239 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224815+0000", 2023-10-20T03:36:18.239 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053536399999999998 2023-10-20T03:36:18.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.240 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.240 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.240 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224833+0000", 2023-10-20T03:36:18.240 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8280000000000001e-05 2023-10-20T03:36:18.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.241 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.241 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.241 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224845+0000", 2023-10-20T03:36:18.241 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1942e-05 2023-10-20T03:36:18.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.242 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.242 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.242 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224989+0000", 2023-10-20T03:36:18.242 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000144368 2023-10-20T03:36:18.242 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.243 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.243 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.225011+0000", 2023-10-20T03:36:18.243 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1953000000000001e-05 2023-10-20T03:36:18.244 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.244 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.244 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.244 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.244 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-10-20T03:36:18.245 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 1 2023-10-20T03:36:18.245 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:18.245 INFO:tasks.rados.rados.0.smithi170.stdout:341: done (2 left) 2023-10-20T03:36:18.246 INFO:tasks.rados.rados.0.smithi170.stdout:342: snap_remove snap 29 2023-10-20T03:36:18.248 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:18.247+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:18.250 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-10-20T03:36:18.349 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:18.344+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:18.420 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:18.420 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:36:18.421 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:36:18.421 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:36:18.421 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:18.435 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-10-20T03:36:18.450 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:18.444+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:18.551 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:18.548+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:18.562 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:18.562 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:36:18.562 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:36:18.563 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:36:18.563 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.563 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:316 3.5s0 3:aed1dba0:::smithi17039300-29:head [append 0~0] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:36:18.563 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.364850+0000", 2023-10-20T03:36:18.563 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 53.195405804000004, 2023-10-20T03:36:18.564 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.016187792999999999, 2023-10-20T03:36:18.564 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.564 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:18.564 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:18.564 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:18.564 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:18.565 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 316 2023-10-20T03:36:18.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.565 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.565 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.565 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364850+0000", 2023-10-20T03:36:18.566 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.566 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.566 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.566 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364850+0000", 2023-10-20T03:36:18.567 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.567 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.567 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.567 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364852+0000", 2023-10-20T03:36:18.568 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0260000000000002e-06 2023-10-20T03:36:18.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.568 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.568 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.568 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364861+0000", 2023-10-20T03:36:18.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.5899999999999997e-06 2023-10-20T03:36:18.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.569 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.569 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.569 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364880+0000", 2023-10-20T03:36:18.569 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9245999999999998e-05 2023-10-20T03:36:18.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.570 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.570 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.570 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364892+0000", 2023-10-20T03:36:18.570 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.184e-05 2023-10-20T03:36:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364987+0000", 2023-10-20T03:36:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4752000000000004e-05 2023-10-20T03:36:18.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.572 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2023-10-20T03:36:18.572 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364997+0000", 2023-10-20T03:36:18.572 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0319999999999999e-05 2023-10-20T03:36:18.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.572 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.573 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.573 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.368108+0000", 2023-10-20T03:36:18.573 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0031111200000000002 2023-10-20T03:36:18.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.573 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.574 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:18.574 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.368196+0000", 2023-10-20T03:36:18.574 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7773000000000001e-05 2023-10-20T03:36:18.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.574 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.574 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.368815+0000", 2023-10-20T03:36:18.575 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00061905900000000004 2023-10-20T03:36:18.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.575 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.575 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.575 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370813+0000", 2023-10-20T03:36:18.576 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0019981780000000002 2023-10-20T03:36:18.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.576 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.576 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:18.576 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.380943+0000", 2023-10-20T03:36:18.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010129487 2023-10-20T03:36:18.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.577 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.577 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.577 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.381037+0000", 2023-10-20T03:36:18.577 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4401999999999995e-05 2023-10-20T03:36:18.578 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.578 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.578 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.578 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.579 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:317 3.5s0 3:aed1dba0:::smithi17039300-29:head [rollback 9] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:36:18.579 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.364924+0000", 2023-10-20T03:36:18.579 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 53.195330945999999, 2023-10-20T03:36:18.579 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017874489, 2023-10-20T03:36:18.579 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.579 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:18.580 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:18.580 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:18.580 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:18.580 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 317 2023-10-20T03:36:18.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.581 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.581 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.581 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.581 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364924+0000", 2023-10-20T03:36:18.581 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.582 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.582 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.582 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364924+0000", 2023-10-20T03:36:18.582 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.583 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.583 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364925+0000", 2023-10-20T03:36:18.583 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8599999999999998e-07 2023-10-20T03:36:18.583 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.584 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.584 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.584 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364929+0000", 2023-10-20T03:36:18.584 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7309999999999999e-06 2023-10-20T03:36:18.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.585 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.585 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.585 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364940+0000", 2023-10-20T03:36:18.585 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1236999999999999e-05 2023-10-20T03:36:18.585 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.585 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.586 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.586 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364947+0000", 2023-10-20T03:36:18.586 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0469999999999996e-06 2023-10-20T03:36:18.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.586 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.587 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.587 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365101+0000", 2023-10-20T03:36:18.587 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015391 2023-10-20T03:36:18.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.587 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:18.588 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365110+0000", 2023-10-20T03:36:18.588 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2e-06 2023-10-20T03:36:18.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.588 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.588 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.589 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.369303+0000", 2023-10-20T03:36:18.589 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041924149999999997 2023-10-20T03:36:18.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.589 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.589 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:18.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.369380+0000", 2023-10-20T03:36:18.590 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.716e-05 2023-10-20T03:36:18.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.590 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.590 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.590 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.369806+0000", 2023-10-20T03:36:18.591 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00042586099999999998 2023-10-20T03:36:18.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.591 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.591 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.372757+0000", 2023-10-20T03:36:18.592 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0029512430000000001 2023-10-20T03:36:18.592 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.592 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.592 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:18.592 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.382729+0000", 2023-10-20T03:36:18.592 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099724180000000003 2023-10-20T03:36:18.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.593 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.593 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.593 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.382799+0000", 2023-10-20T03:36:18.593 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9680999999999999e-05 2023-10-20T03:36:18.594 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.594 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.594 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.594 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.595 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:318 3.5s0 3:aed1dba0:::smithi17039300-29:head [append 0~0] snapc a=[a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-10-20T03:36:18.595 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:36:18.595 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 53.195270792999999, 2023-10-20T03:36:18.595 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018045379, 2023-10-20T03:36:18.595 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.596 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:18.596 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:18.596 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:18.596 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:18.596 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 318 2023-10-20T03:36:18.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.597 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.597 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.597 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:36:18.597 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.598 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.598 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.598 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.598 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:36:18.598 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.599 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.599 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.599 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364985+0000", 2023-10-20T03:36:18.599 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8899999999999999e-07 2023-10-20T03:36:18.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.600 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.600 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.600 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364989+0000", 2023-10-20T03:36:18.600 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7079999999999999e-06 2023-10-20T03:36:18.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.601 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.601 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.601 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.364998+0000", 2023-10-20T03:36:18.601 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2229999999999996e-06 2023-10-20T03:36:18.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.602 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.602 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.602 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365004+0000", 2023-10-20T03:36:18.602 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1999999999999999e-06 2023-10-20T03:36:18.602 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.602 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.603 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.603 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365157+0000", 2023-10-20T03:36:18.603 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015222799999999999 2023-10-20T03:36:18.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.604 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:18.604 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.365166+0000", 2023-10-20T03:36:18.604 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6739999999999992e-06 2023-10-20T03:36:18.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.604 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.605 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.605 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370187+0000", 2023-10-20T03:36:18.605 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0050205279999999998 2023-10-20T03:36:18.605 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.605 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.605 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:18.606 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370256+0000", 2023-10-20T03:36:18.606 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9151000000000003e-05 2023-10-20T03:36:18.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.606 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.606 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.607 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.370618+0000", 2023-10-20T03:36:18.607 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00036223100000000002 2023-10-20T03:36:18.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.607 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.607 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.608 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.375514+0000", 2023-10-20T03:36:18.608 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0048961220000000001 2023-10-20T03:36:18.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.608 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.608 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:18.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.382954+0000", 2023-10-20T03:36:18.609 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0074393280000000003 2023-10-20T03:36:18.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.609 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.609 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.609 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.383030+0000", 2023-10-20T03:36:18.610 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6396999999999994e-05 2023-10-20T03:36:18.610 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.610 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.610 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.611 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:18.611 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:36:18.611 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.171589685000001, 2023-10-20T03:36:18.611 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028147078999999998, 2023-10-20T03:36:18.611 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.612 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.612 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.612 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.612 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.612 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:36:18.613 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.613 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.613 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.613 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:36:18.613 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.614 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.614 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.614 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388666+0000", 2023-10-20T03:36:18.614 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4600000000000004e-07 2023-10-20T03:36:18.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.615 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.615 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.615 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388785+0000", 2023-10-20T03:36:18.615 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000118604 2023-10-20T03:36:18.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.616 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.616 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.616 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388827+0000", 2023-10-20T03:36:18.616 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2404000000000003e-05 2023-10-20T03:36:18.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.617 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.617 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.617 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388843+0000", 2023-10-20T03:36:18.617 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5894000000000001e-05 2023-10-20T03:36:18.617 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.618 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.618 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.618 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.389002+0000", 2023-10-20T03:36:18.618 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000158383 2023-10-20T03:36:18.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.619 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.619 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.389019+0000", 2023-10-20T03:36:18.619 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7266e-05 2023-10-20T03:36:18.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.619 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.620 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.620 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416738+0000", 2023-10-20T03:36:18.620 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.027718779999999998 2023-10-20T03:36:18.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.620 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.620 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.621 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416813+0000", 2023-10-20T03:36:18.621 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5102e-05 2023-10-20T03:36:18.621 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.621 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.621 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.622 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.622 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:340 3.5s0 3:aed1dba0:::smithi17039300-29:head [create,setxattr nWX_d[W\\dXUWOgbXndrSbcTU`mgjR_]o\\nNcXpin (1403) in=40b,setxattr TSP[ddnm_WR]nh]QhSXpkpmmWeOP^b]baemT^Uqr (630) in=40b,setxattr R[QU^NWYrjfarmhlWmZlUkP\\ipWan`SnqZOX`VWh (806) in=40b,setxattr ]hV\\^[PQZO]n^abhilRfldoUhgtl\\\\oel`mgahd_ (3368) in=40b,setxattr OYc`XQ\\VjZfQ`]OsbemUo_mfVUSc^pNo\\[\\ldrhg (1087) in=40b,setxattr `pZtTmno_eqZOONfatgZ_`nnhemaaQWiteO]dc[V (175) in=40b,setxattr PdQa`feQapkrnjfOUYUeqsOt_hrsg\\noqrcV]pSh (981) in=40b,setxattr bk_QpaSsdlQfT`oa[pTQS_[ad]isfYm^tQNdhSiW (1718) in=40b,setxattr Ys`Y_UZ^PXhpgS`VS]`eVgkcmT^QW[aWscssRRSr (10033) in=40b,setxattr [jd]Q\\qemWTVcc`hiZnsrqrggetajZRa]\\_llS`h (1367) in=40b,setxattr `UajoeZh]tRSf\\[nnVat^TjN^_q`ooQ_br`itgZm (244) in=40b,setxattr Tng[WcWa_dsnRXtZNa^`fle`gQ___mar\\fgRsZVY (665) in=40b,setxattr nogmhepVZfQtnZc^cZSeaaogjklZkrff_Utg`UR^ (15856) in=40b,setxattr gfqSsQY[][rrac[gmaQrTSacNWttTTXhanmRa^fa (1998) in=40b,setxattr V`Y[ddb`_dfep[_]oUoTqQj\\UW[gkesnlgNPYtPa (1438) in=40b,setxattr YgoRVqUOj_mXPXsq^PdjaaolmQsTsTsZpaddktl[ (1400) in=40b,setxattr smn[QVPVkfVs]oTip]jtirOtU\\Pa\\Ym^poZgfZaq (1263) in=40b,setxattr TQSmSs^Z]\\rdosO_a^TOU^[p`jkQb`dXXh]s`Stq (909) in=40b,setxattr faPhpqVXmWh`VeNZo[b^[kYl^[[rYYPRrk_[ON`e (14664) in=40b,setxattr ksO_UmqqQefe[Q\\njt[W_roWeei`]Sn^TV]O^iSW (1963) in=40b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:18.622 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.397436+0000", 2023-10-20T03:36:18.622 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.162818989000002, 2023-10-20T03:36:18.623 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.024477860000000001, 2023-10-20T03:36:18.623 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.623 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:18.623 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:18.623 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:18.623 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:18.624 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 340 2023-10-20T03:36:18.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.624 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.624 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.624 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397436+0000", 2023-10-20T03:36:18.625 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.625 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.625 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.625 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397436+0000", 2023-10-20T03:36:18.626 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.626 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.626 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.626 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397439+0000", 2023-10-20T03:36:18.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6929999999999999e-06 2023-10-20T03:36:18.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.627 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.627 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397608+0000", 2023-10-20T03:36:18.627 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016918 2023-10-20T03:36:18.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.628 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.628 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.628 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397637+0000", 2023-10-20T03:36:18.628 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9057e-05 2023-10-20T03:36:18.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.629 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.629 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.629 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397653+0000", 2023-10-20T03:36:18.629 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5430999999999998e-05 2023-10-20T03:36:18.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.630 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.630 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.630 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397749+0000", 2023-10-20T03:36:18.630 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6694000000000006e-05 2023-10-20T03:36:18.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.631 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.631 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:18.631 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.397877+0000", 2023-10-20T03:36:18.631 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012762599999999999 2023-10-20T03:36:18.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.632 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.632 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.399009+0000", 2023-10-20T03:36:18.632 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011323640000000001 2023-10-20T03:36:18.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.633 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.633 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416407+0000", 2023-10-20T03:36:18.633 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017398034999999999 2023-10-20T03:36:18.633 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.633 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.634 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:18.634 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.421802+0000", 2023-10-20T03:36:18.634 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0053944320000000002 2023-10-20T03:36:18.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.634 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.635 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.635 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.421914+0000", 2023-10-20T03:36:18.635 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000112348 2023-10-20T03:36:18.635 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.635 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.635 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.636 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.636 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:18.636 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.401616+0000", 2023-10-20T03:36:18.636 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.158639233999999, 2023-10-20T03:36:18.637 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015282946, 2023-10-20T03:36:18.637 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.637 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.637 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.637 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.638 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401616+0000", 2023-10-20T03:36:18.638 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.638 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.638 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.639 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401616+0000", 2023-10-20T03:36:18.639 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.639 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.639 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401617+0000", 2023-10-20T03:36:18.640 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.105e-06 2023-10-20T03:36:18.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.640 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.640 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.640 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401748+0000", 2023-10-20T03:36:18.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013050599999999999 2023-10-20T03:36:18.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.641 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.641 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401765+0000", 2023-10-20T03:36:18.641 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7450999999999999e-05 2023-10-20T03:36:18.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.642 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.642 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.642 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401774+0000", 2023-10-20T03:36:18.642 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9760000000000001e-06 2023-10-20T03:36:18.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.643 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.643 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.643 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404428+0000", 2023-10-20T03:36:18.643 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00265409 2023-10-20T03:36:18.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.644 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.644 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.644 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404442+0000", 2023-10-20T03:36:18.644 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3602e-05 2023-10-20T03:36:18.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.645 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.645 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.645 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416847+0000", 2023-10-20T03:36:18.645 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012404837 2023-10-20T03:36:18.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.646 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.646 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416899+0000", 2023-10-20T03:36:18.646 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2379000000000001e-05 2023-10-20T03:36:18.646 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.647 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.647 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.647 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.647 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=53, reqid=client.4570.0:335, at_version=44'20, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:18.647 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.403071+0000", 2023-10-20T03:36:18.648 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.157184233999999, 2023-10-20T03:36:18.648 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014226947, 2023-10-20T03:36:18.648 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.648 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.648 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.648 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.649 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.649 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403071+0000", 2023-10-20T03:36:18.649 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.649 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.650 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.650 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403071+0000", 2023-10-20T03:36:18.650 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.650 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.651 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.651 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403072+0000", 2023-10-20T03:36:18.651 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2299999999999999e-07 2023-10-20T03:36:18.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.651 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.651 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.652 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403107+0000", 2023-10-20T03:36:18.652 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4900000000000001e-05 2023-10-20T03:36:18.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.652 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.652 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.653 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403125+0000", 2023-10-20T03:36:18.653 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8386999999999999e-05 2023-10-20T03:36:18.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.653 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.653 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.653 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403138+0000", 2023-10-20T03:36:18.654 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2979999999999999e-05 2023-10-20T03:36:18.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.654 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.654 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.654 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413715+0000", 2023-10-20T03:36:18.655 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010576538999999999 2023-10-20T03:36:18.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.655 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.655 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.655 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.413733+0000", 2023-10-20T03:36:18.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8357000000000001e-05 2023-10-20T03:36:18.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.656 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.656 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.656 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417237+0000", 2023-10-20T03:36:18.656 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0035037689999999999 2023-10-20T03:36:18.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.657 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.657 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.657 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417298+0000", 2023-10-20T03:36:18.657 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1092000000000004e-05 2023-10-20T03:36:18.658 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.658 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.658 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.658 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.658 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.658 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 44/31 ECSubWrite(tid=54, reqid=client.4570.0:336, at_version=44'21, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:18.661 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.405096+0000", 2023-10-20T03:36:18.661 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.155159177000002, 2023-10-20T03:36:18.661 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012879895000000001, 2023-10-20T03:36:18.662 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.662 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.662 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.662 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.663 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405096+0000", 2023-10-20T03:36:18.663 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.663 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.664 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405096+0000", 2023-10-20T03:36:18.664 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.664 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.664 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.664 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.664 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405097+0000", 2023-10-20T03:36:18.665 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.035e-06 2023-10-20T03:36:18.665 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.665 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.665 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.665 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405121+0000", 2023-10-20T03:36:18.666 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4097e-05 2023-10-20T03:36:18.666 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.666 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.666 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.666 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405177+0000", 2023-10-20T03:36:18.667 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.5220000000000003e-05 2023-10-20T03:36:18.667 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.667 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.667 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.667 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405192+0000", 2023-10-20T03:36:18.667 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.579e-05 2023-10-20T03:36:18.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.668 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.668 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.668 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416447+0000", 2023-10-20T03:36:18.668 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011254188 2023-10-20T03:36:18.669 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.669 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.669 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416463+0000", 2023-10-20T03:36:18.669 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6441000000000001e-05 2023-10-20T03:36:18.669 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.670 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.670 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.670 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417913+0000", 2023-10-20T03:36:18.670 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0014495809999999999 2023-10-20T03:36:18.670 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.671 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.671 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.671 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.417976+0000", 2023-10-20T03:36:18.671 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3542999999999998e-05 2023-10-20T03:36:18.671 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.672 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.672 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.672 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.672 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.672 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 44/31 ECSubWriteReply(tid=22, last_complete=44'12, committed=1, applied=1))", 2023-10-20T03:36:18.672 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.406456+0000", 2023-10-20T03:36:18.673 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.153799329000002, 2023-10-20T03:36:18.673 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010387031999999999, 2023-10-20T03:36:18.673 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.673 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.673 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.674 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.674 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406456+0000", 2023-10-20T03:36:18.674 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.675 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.675 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406456+0000", 2023-10-20T03:36:18.675 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.675 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.676 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.676 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406457+0000", 2023-10-20T03:36:18.676 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6199999999999996e-07 2023-10-20T03:36:18.676 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.676 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.677 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.677 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406462+0000", 2023-10-20T03:36:18.677 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.2889999999999997e-06 2023-10-20T03:36:18.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.677 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.677 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.678 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406479+0000", 2023-10-20T03:36:18.678 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6963e-05 2023-10-20T03:36:18.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.678 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.678 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.679 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406488+0000", 2023-10-20T03:36:18.679 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.6200000000000005e-06 2023-10-20T03:36:18.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.679 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.679 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.680 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416476+0000", 2023-10-20T03:36:18.680 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099880660000000003 2023-10-20T03:36:18.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.680 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.680 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.680 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416843+0000", 2023-10-20T03:36:18.681 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000367232 2023-10-20T03:36:18.681 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.681 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.681 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.682 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.682 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 44/31 ECSubWriteReply(tid=22, last_complete=44'12, committed=1, applied=1))", 2023-10-20T03:36:18.682 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.406529+0000", 2023-10-20T03:36:18.682 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.153726075000002, 2023-10-20T03:36:18.682 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010245637, 2023-10-20T03:36:18.683 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.683 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.683 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.683 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.683 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.683 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406529+0000", 2023-10-20T03:36:18.684 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.684 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.684 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.684 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406529+0000", 2023-10-20T03:36:18.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.685 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.685 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.685 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.685 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406530+0000", 2023-10-20T03:36:18.685 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.9599999999999999e-07 2023-10-20T03:36:18.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.686 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.686 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.686 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406541+0000", 2023-10-20T03:36:18.686 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1208e-05 2023-10-20T03:36:18.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.687 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.687 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406557+0000", 2023-10-20T03:36:18.687 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5563999999999999e-05 2023-10-20T03:36:18.688 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.688 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.688 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.688 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406566+0000", 2023-10-20T03:36:18.688 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.0769999999999997e-06 2023-10-20T03:36:18.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.689 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.689 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.689 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416751+0000", 2023-10-20T03:36:18.689 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010184794 2023-10-20T03:36:18.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.690 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.690 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.690 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.416775+0000", 2023-10-20T03:36:18.690 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4298e-05 2023-10-20T03:36:18.690 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.691 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.691 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.691 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.691 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.ds4 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:36:18.692 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.393394+0000", 2023-10-20T03:36:18.692 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.166860886999999, 2023-10-20T03:36:18.692 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.039046538999999998, 2023-10-20T03:36:18.692 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.692 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.692 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.693 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.693 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.693 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393394+0000", 2023-10-20T03:36:18.693 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.694 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.694 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393394+0000", 2023-10-20T03:36:18.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.694 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.393395+0000", 2023-10-20T03:36:18.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.16e-07 2023-10-20T03:36:18.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394217+0000", 2023-10-20T03:36:18.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00082179599999999996 2023-10-20T03:36:18.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394258+0000", 2023-10-20T03:36:18.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0546000000000001e-05 2023-10-20T03:36:18.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394283+0000", 2023-10-20T03:36:18.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5707e-05 2023-10-20T03:36:18.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.699 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394593+0000", 2023-10-20T03:36:18.699 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031000800000000002 2023-10-20T03:36:18.699 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.699 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.699 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.699 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394630+0000", 2023-10-20T03:36:18.700 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.6885999999999999e-05 2023-10-20T03:36:18.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.700 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.700 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432353+0000", 2023-10-20T03:36:18.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.037723055999999998 2023-10-20T03:36:18.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432441+0000", 2023-10-20T03:36:18.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.7724000000000002e-05 2023-10-20T03:36:18.702 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.702 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.702 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.702 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.703 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:392 3.5s0 3:aed1dba0:::smithi17039300-29:head [stat,delete] snapc f=[f,d,b,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:36:18.703 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.382023+0000", 2023-10-20T03:36:18.703 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 47.178232426000001, 2023-10-20T03:36:18.703 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019348521, 2023-10-20T03:36:18.703 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.704 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:18.704 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:18.704 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:18.704 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:18.704 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 392 2023-10-20T03:36:18.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.705 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.705 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.705 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382023+0000", 2023-10-20T03:36:18.705 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382023+0000", 2023-10-20T03:36:18.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.707 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.707 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382026+0000", 2023-10-20T03:36:18.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3629999999999998e-06 2023-10-20T03:36:18.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382040+0000", 2023-10-20T03:36:18.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3913e-05 2023-10-20T03:36:18.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.709 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.709 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.709 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382075+0000", 2023-10-20T03:36:18.709 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.4678999999999998e-05 2023-10-20T03:36:18.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.710 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.710 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382093+0000", 2023-10-20T03:36:18.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7779999999999999e-05 2023-10-20T03:36:18.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.711 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.711 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382244+0000", 2023-10-20T03:36:18.711 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015178300000000001 2023-10-20T03:36:18.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.712 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2023-10-20T03:36:18.712 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.382257+0000", 2023-10-20T03:36:18.712 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.288e-05 2023-10-20T03:36:18.712 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.712 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.713 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.387421+0000", 2023-10-20T03:36:18.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0051631560000000003 2023-10-20T03:36:18.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.713 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:18.714 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.387619+0000", 2023-10-20T03:36:18.714 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000198712 2023-10-20T03:36:18.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.714 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.715 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.389154+0000", 2023-10-20T03:36:18.715 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001535193 2023-10-20T03:36:18.715 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.715 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.715 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.716 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.393045+0000", 2023-10-20T03:36:18.716 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0038909639999999998 2023-10-20T03:36:18.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.716 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.716 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:18.717 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.401188+0000", 2023-10-20T03:36:18.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0081430219999999998 2023-10-20T03:36:18.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.717 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.717 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.401371+0000", 2023-10-20T03:36:18.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018307600000000001 2023-10-20T03:36:18.718 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.718 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.718 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.719 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.719 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.3s1 71/56 ECSubWrite(tid=63, reqid=client.4570.0:575, at_version=71'46, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:18.719 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:52.983783+0000", 2023-10-20T03:36:18.719 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 25.576472216999999, 2023-10-20T03:36:18.719 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011281321, 2023-10-20T03:36:18.719 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.720 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.720 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.720 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.983783+0000", 2023-10-20T03:36:18.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.721 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.983783+0000", 2023-10-20T03:36:18.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.722 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.722 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.722 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.983784+0000", 2023-10-20T03:36:18.722 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3179999999999999e-06 2023-10-20T03:36:18.723 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.723 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.723 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.723 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984072+0000", 2023-10-20T03:36:18.723 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00028775600000000001 2023-10-20T03:36:18.724 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.724 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.724 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.724 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984099+0000", 2023-10-20T03:36:18.724 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7019999999999999e-05 2023-10-20T03:36:18.724 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.725 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.725 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.725 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984114+0000", 2023-10-20T03:36:18.725 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5184999999999999e-05 2023-10-20T03:36:18.725 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.726 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.726 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.726 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984197+0000", 2023-10-20T03:36:18.726 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2096999999999997e-05 2023-10-20T03:36:18.726 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.727 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.727 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.727 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.984209+0000", 2023-10-20T03:36:18.727 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2031999999999999e-05 2023-10-20T03:36:18.727 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.727 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.728 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.728 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.994991+0000", 2023-10-20T03:36:18.728 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010782862000000001 2023-10-20T03:36:18.728 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.728 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.729 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.729 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:52.995064+0000", 2023-10-20T03:36:18.729 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3051000000000002e-05 2023-10-20T03:36:18.729 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.729 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.729 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.730 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.730 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.730 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.as6 82/56 ECSubWrite(tid=36, reqid=client.4570.0:664, at_version=82'39, trim_to=0'0, roll_forward_to=81'35))", 2023-10-20T03:36:18.730 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:03.785456+0000", 2023-10-20T03:36:18.730 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 14.774799196, 2023-10-20T03:36:18.731 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0098676900000000001, 2023-10-20T03:36:18.731 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.731 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.731 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.731 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.732 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.732 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:03.785456+0000", 2023-10-20T03:36:18.732 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.732 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.732 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.732 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.733 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:03.785456+0000", 2023-10-20T03:36:18.733 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.733 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.733 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.733 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.734 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:03.785458+0000", 2023-10-20T03:36:18.734 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6700000000000001e-06 2023-10-20T03:36:18.734 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.734 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.734 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.735 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:03.785512+0000", 2023-10-20T03:36:18.735 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.4123999999999999e-05 2023-10-20T03:36:18.735 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.735 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.735 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.736 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:03.785539+0000", 2023-10-20T03:36:18.736 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7294999999999999e-05 2023-10-20T03:36:18.736 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.736 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.736 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.736 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:03.785562+0000", 2023-10-20T03:36:18.737 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.2595999999999998e-05 2023-10-20T03:36:18.737 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.737 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.737 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.737 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:03.789487+0000", 2023-10-20T03:36:18.738 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0039253609999999996 2023-10-20T03:36:18.738 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.738 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.738 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.738 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:03.789512+0000", 2023-10-20T03:36:18.739 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4953e-05 2023-10-20T03:36:18.739 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.739 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.739 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.739 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:03.795263+0000", 2023-10-20T03:36:18.740 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0057505209999999998 2023-10-20T03:36:18.740 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.740 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.740 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.740 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:03.795324+0000", 2023-10-20T03:36:18.740 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.1169999999999999e-05 2023-10-20T03:36:18.741 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.741 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.741 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.741 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.741 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.742 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:784 3.5s0 3:a2413f50:::smithi17039300-48:head [copy-from ver 0] snapc 39=[39,38,37,36] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2023-10-20T03:36:18.742 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:13.124094+0000", 2023-10-20T03:36:18.742 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.4361609619999998, 2023-10-20T03:36:18.742 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.112764683, 2023-10-20T03:36:18.742 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.743 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:36:18.743 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:18.743 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:18.743 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:18.743 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 784 2023-10-20T03:36:18.743 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.744 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.744 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.744 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.744 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124094+0000", 2023-10-20T03:36:18.744 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.745 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.745 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.745 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.745 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124094+0000", 2023-10-20T03:36:18.745 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.746 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.746 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.746 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.746 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124098+0000", 2023-10-20T03:36:18.746 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8500000000000004e-06 2023-10-20T03:36:18.747 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.747 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.747 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.747 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124116+0000", 2023-10-20T03:36:18.747 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7269e-05 2023-10-20T03:36:18.747 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.748 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.748 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.748 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124161+0000", 2023-10-20T03:36:18.748 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.5346000000000003e-05 2023-10-20T03:36:18.748 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.749 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.749 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.749 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124187+0000", 2023-10-20T03:36:18.749 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5639e-05 2023-10-20T03:36:18.749 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.749 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.750 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.750 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124348+0000", 2023-10-20T03:36:18.750 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000161531 2023-10-20T03:36:18.750 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.750 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.751 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "op must wait for map", 2023-10-20T03:36:18.751 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124364+0000", 2023-10-20T03:36:18.751 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6073e-05 2023-10-20T03:36:18.751 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.751 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.751 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.752 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.137867+0000", 2023-10-20T03:36:18.752 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013503345 2023-10-20T03:36:18.752 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.752 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.752 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:18.753 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.138308+0000", 2023-10-20T03:36:18.753 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00044069299999999998 2023-10-20T03:36:18.753 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.753 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.753 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.754 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.236859+0000", 2023-10-20T03:36:18.754 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.098550937000000005 2023-10-20T03:36:18.754 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.754 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.754 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.754 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.755 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.755 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:785 3.5s0 3:a2413f50:::smithi17039300-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2023-10-20T03:36:18.755 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:13.124245+0000", 2023-10-20T03:36:18.755 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.4360104140000001, 2023-10-20T03:36:18.755 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.113214232, 2023-10-20T03:36:18.756 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.756 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:36:18.756 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:18.756 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:18.756 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:18.757 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 785 2023-10-20T03:36:18.757 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.757 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.757 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.757 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.757 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124245+0000", 2023-10-20T03:36:18.758 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.758 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.758 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.758 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.758 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124245+0000", 2023-10-20T03:36:18.759 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.759 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.759 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.759 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.759 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124247+0000", 2023-10-20T03:36:18.759 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.79e-06 2023-10-20T03:36:18.760 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.760 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.760 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.760 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124257+0000", 2023-10-20T03:36:18.760 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0360000000000001e-05 2023-10-20T03:36:18.761 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.761 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.761 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.761 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124287+0000", 2023-10-20T03:36:18.761 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9589000000000001e-05 2023-10-20T03:36:18.761 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.762 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.762 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.762 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124306+0000", 2023-10-20T03:36:18.762 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9715999999999999e-05 2023-10-20T03:36:18.762 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.763 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.763 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.763 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124505+0000", 2023-10-20T03:36:18.763 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000198574 2023-10-20T03:36:18.763 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.764 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.764 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:18.764 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.124544+0000", 2023-10-20T03:36:18.764 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9412000000000001e-05 2023-10-20T03:36:18.764 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.764 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.765 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.765 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.138837+0000", 2023-10-20T03:36:18.765 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014292911 2023-10-20T03:36:18.765 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.765 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.766 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2023-10-20T03:36:18.766 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.140606+0000", 2023-10-20T03:36:18.766 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001769119 2023-10-20T03:36:18.766 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.766 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.767 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.767 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.237068+0000", 2023-10-20T03:36:18.767 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.096461950000000005 2023-10-20T03:36:18.767 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.767 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.767 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:18.768 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.237255+0000", 2023-10-20T03:36:18.768 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018653499999999999 2023-10-20T03:36:18.768 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.768 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.768 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.769 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.237459+0000", 2023-10-20T03:36:18.769 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00020427600000000001 2023-10-20T03:36:18.769 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.769 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.769 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.769 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.770 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.770 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 91/31 ECSubWrite(tid=93, reqid=client.4570.0:781, at_version=91'71, trim_to=0'0, roll_forward_to=88'69))", 2023-10-20T03:36:18.770 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:13.135014+0000", 2023-10-20T03:36:18.770 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 5.425241701, 2023-10-20T03:36:18.770 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010016412000000001, 2023-10-20T03:36:18.771 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.771 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.771 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.771 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.771 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.772 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.135014+0000", 2023-10-20T03:36:18.772 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.772 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.772 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.772 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.772 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.135014+0000", 2023-10-20T03:36:18.773 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.773 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.773 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.773 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.773 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.135014+0000", 2023-10-20T03:36:18.774 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4600000000000004e-07 2023-10-20T03:36:18.774 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.774 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.774 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.774 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.135068+0000", 2023-10-20T03:36:18.775 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3298999999999999e-05 2023-10-20T03:36:18.775 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.775 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.775 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.775 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.135101+0000", 2023-10-20T03:36:18.775 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3426000000000002e-05 2023-10-20T03:36:18.776 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.776 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.776 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.776 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.135126+0000", 2023-10-20T03:36:18.776 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5262e-05 2023-10-20T03:36:18.777 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.777 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.777 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.777 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.135390+0000", 2023-10-20T03:36:18.777 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00026326600000000001 2023-10-20T03:36:18.777 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.778 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.778 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.778 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.135424+0000", 2023-10-20T03:36:18.778 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3985000000000001e-05 2023-10-20T03:36:18.778 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.779 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.779 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.779 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.144941+0000", 2023-10-20T03:36:18.779 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0095173710000000002 2023-10-20T03:36:18.779 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.780 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.780 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.780 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:13.145030+0000", 2023-10-20T03:36:18.780 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9056999999999998e-05 2023-10-20T03:36:18.780 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.780 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.781 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.781 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.781 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.781 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 92/31 ECSubWrite(tid=97, reqid=osd.14.0:97, at_version=92'75, trim_to=0'0, roll_forward_to=91'73))", 2023-10-20T03:36:18.781 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:14.151691+0000", 2023-10-20T03:36:18.782 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 4.4085640020000003, 2023-10-20T03:36:18.782 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0098453150000000003, 2023-10-20T03:36:18.782 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.782 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.782 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.783 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.783 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.783 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151691+0000", 2023-10-20T03:36:18.783 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.783 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.784 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.784 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151691+0000", 2023-10-20T03:36:18.784 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.784 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.784 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.785 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.785 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151692+0000", 2023-10-20T03:36:18.785 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9700000000000002e-07 2023-10-20T03:36:18.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.785 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.786 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151739+0000", 2023-10-20T03:36:18.786 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7252000000000001e-05 2023-10-20T03:36:18.786 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.786 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.786 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.787 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151772+0000", 2023-10-20T03:36:18.787 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.3018999999999997e-05 2023-10-20T03:36:18.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.787 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.787 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.788 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.151796+0000", 2023-10-20T03:36:18.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3606000000000001e-05 2023-10-20T03:36:18.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.788 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.788 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.788 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.154086+0000", 2023-10-20T03:36:18.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0022907819999999999 2023-10-20T03:36:18.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.789 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.789 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.154116+0000", 2023-10-20T03:36:18.790 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.9309e-05 2023-10-20T03:36:18.790 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.790 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.790 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.161440+0000", 2023-10-20T03:36:18.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0073240429999999997 2023-10-20T03:36:18.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.791 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.791 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:14.161537+0000", 2023-10-20T03:36:18.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.6906999999999999e-05 2023-10-20T03:36:18.792 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.792 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.792 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.793 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 96/31 ECSubWrite(tid=102, reqid=client.4570.0:824, at_version=96'81, trim_to=0'0, roll_forward_to=96'79))", 2023-10-20T03:36:18.793 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:18.224278+0000", 2023-10-20T03:36:18.793 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.33597763600000002, 2023-10-20T03:36:18.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014201271, 2023-10-20T03:36:18.793 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.794 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.794 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224278+0000", 2023-10-20T03:36:18.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.795 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.795 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224278+0000", 2023-10-20T03:36:18.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.796 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.796 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224279+0000", 2023-10-20T03:36:18.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5659999999999999e-06 2023-10-20T03:36:18.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.797 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.797 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.797 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224815+0000", 2023-10-20T03:36:18.797 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00053536399999999998 2023-10-20T03:36:18.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.798 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.798 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.798 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224833+0000", 2023-10-20T03:36:18.798 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8280000000000001e-05 2023-10-20T03:36:18.798 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.798 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.799 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.799 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224845+0000", 2023-10-20T03:36:18.799 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1942e-05 2023-10-20T03:36:18.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.799 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.800 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.800 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.224989+0000", 2023-10-20T03:36:18.800 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000144368 2023-10-20T03:36:18.800 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.800 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.800 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.801 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.225011+0000", 2023-10-20T03:36:18.801 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.1953000000000001e-05 2023-10-20T03:36:18.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.801 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.802 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.238439+0000", 2023-10-20T03:36:18.802 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013427774999999999 2023-10-20T03:36:18.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.802 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.802 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.803 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.238479+0000", 2023-10-20T03:36:18.803 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.0022999999999999e-05 2023-10-20T03:36:18.803 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.803 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.803 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.804 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.804 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.2s4 96/31 ECSubWrite(tid=105, reqid=client.4570.0:826, at_version=96'83, trim_to=0'0, roll_forward_to=96'82))", 2023-10-20T03:36:18.804 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:36:18.265152+0000", 2023-10-20T03:36:18.804 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 0.29510340899999998, 2023-10-20T03:36:18.804 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0099068260000000005, 2023-10-20T03:36:18.805 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:18.805 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:18.805 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:18.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.805 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:18.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.265152+0000", 2023-10-20T03:36:18.806 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.806 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:18.806 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.265152+0000", 2023-10-20T03:36:18.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:18.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:18.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.265153+0000", 2023-10-20T03:36:18.808 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.4300000000000002e-07 2023-10-20T03:36:18.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.808 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:18.808 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.265374+0000", 2023-10-20T03:36:18.808 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022130300000000001 2023-10-20T03:36:18.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.809 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.809 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:18.809 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.265391+0000", 2023-10-20T03:36:18.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6646e-05 2023-10-20T03:36:18.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.810 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:18.810 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.265400+0000", 2023-10-20T03:36:18.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2269999999999999e-06 2023-10-20T03:36:18.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.811 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.811 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:18.811 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.265472+0000", 2023-10-20T03:36:18.811 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1631999999999995e-05 2023-10-20T03:36:18.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:18.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.265485+0000", 2023-10-20T03:36:18.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2991e-05 2023-10-20T03:36:18.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:18.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.275027+0000", 2023-10-20T03:36:18.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0095421499999999992 2023-10-20T03:36:18.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:18.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:18.814 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:18.814 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:36:18.275059+0000", 2023-10-20T03:36:18.814 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2134000000000001e-05 2023-10-20T03:36:18.814 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.814 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.815 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.815 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:18.815 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:18.815 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:18.817 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-10-20T03:36:18.817 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:18.812+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:18.918 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:18.916+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:18.949 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:18.949 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:36:18.949 INFO:teuthology.orchestra.run.smithi033.stdout: "num_ops": 0 2023-10-20T03:36:18.950 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:18.963 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-10-20T03:36:19.018 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:19.016+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:19.083 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:19.083 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:36:19.083 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:36:19.083 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:36:19.083 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:19.098 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-10-20T03:36:19.142 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:19.140+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:19.190 INFO:tasks.rados.rados.0.smithi170.stdout:342: done (2 left) 2023-10-20T03:36:19.190 INFO:tasks.rados.rados.0.smithi170.stdout:343: read oid 17 snap 28 2023-10-20T03:36:19.190 INFO:tasks.rados.rados.0.smithi170.stdout:343: expect (ObjNum 114 snap 19 seq_num 114) 2023-10-20T03:36:19.190 INFO:tasks.rados.rados.0.smithi170.stdout:344: read oid 41 snap -1 2023-10-20T03:36:19.190 INFO:tasks.rados.rados.0.smithi170.stdout:344: expect (ObjNum 116 snap 22 seq_num 116) 2023-10-20T03:36:19.191 INFO:tasks.rados.rados.0.smithi170.stdout:345: rollback oid 5 current snap is 31 2023-10-20T03:36:19.191 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 5 to 28 2023-10-20T03:36:19.191 INFO:tasks.rados.rados.0.smithi170.stdout:339: finishing write tid 1 to smithi17039300-42 2023-10-20T03:36:19.191 INFO:tasks.rados.rados.0.smithi170.stdout:339: finishing write tid 2 to smithi17039300-42 2023-10-20T03:36:19.191 INFO:tasks.rados.rados.0.smithi170.stdout:339: finishing write tid 3 to smithi17039300-42 2023-10-20T03:36:19.192 INFO:tasks.rados.rados.0.smithi170.stdout:346: delete oid 48 current snap is 31 2023-10-20T03:36:19.192 INFO:tasks.rados.rados.0.smithi170.stdout:339: finishing write tid 4 to smithi17039300-42 2023-10-20T03:36:19.192 INFO:tasks.rados.rados.0.smithi170.stdout:339: finishing write tid 5 to smithi17039300-42 2023-10-20T03:36:19.192 INFO:tasks.rados.rados.0.smithi170.stdout:340: finishing rollback tid 0 to smithi17039300-39 2023-10-20T03:36:19.192 INFO:tasks.rados.rados.0.smithi170.stdout:339: finishing write tid 6 to smithi17039300-42 2023-10-20T03:36:19.193 INFO:tasks.rados.rados.0.smithi170.stdout:339: oid 42 updating version 0 to 85 2023-10-20T03:36:19.193 INFO:tasks.rados.rados.0.smithi170.stdout:339: oid 42 version 85 is already newer than 83 2023-10-20T03:36:19.193 INFO:tasks.rados.rados.0.smithi170.stdout:339: oid 42 version 85 is already newer than 84 2023-10-20T03:36:19.193 INFO:tasks.rados.rados.0.smithi170.stdout:339: oid 42 version 85 is already newer than 82 2023-10-20T03:36:19.193 INFO:tasks.rados.rados.0.smithi170.stdout:339: oid 42 version 85 is already newer than 81 2023-10-20T03:36:19.193 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 42 v 85 (ObjNum 140 snap 31 seq_num 140) dirty exists 2023-10-20T03:36:19.194 INFO:tasks.rados.rados.0.smithi170.stdout:339: left oid 42 (ObjNum 140 snap 31 seq_num 140) 2023-10-20T03:36:19.194 INFO:tasks.rados.rados.0.smithi170.stdout:340: finishing rollback tid 1 to smithi17039300-39 2023-10-20T03:36:19.194 INFO:tasks.rados.rados.0.smithi170.stdout:340: finishing rollback tid 2 to smithi17039300-39 2023-10-20T03:36:19.194 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 39 v 88 (ObjNum 129 snap 0 seq_num 3160661384) dirty exists 2023-10-20T03:36:19.216 INFO:tasks.rados.rados.0.smithi170.stdout:345: finishing rollback tid 0 to smithi17039300-5 2023-10-20T03:36:19.217 INFO:tasks.rados.rados.0.smithi170.stdout:339: done (5 left) 2023-10-20T03:36:19.217 INFO:tasks.rados.rados.0.smithi170.stdout:340: done (4 left) 2023-10-20T03:36:19.217 INFO:tasks.rados.rados.0.smithi170.stdout:346: done (3 left) 2023-10-20T03:36:19.217 INFO:tasks.rados.rados.0.smithi170.stdout:347: rollback oid 8 current snap is 31 2023-10-20T03:36:19.218 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 8 to 28 2023-10-20T03:36:19.218 INFO:tasks.rados.rados.0.smithi170.stdout:345: finishing rollback tid 1 to smithi17039300-5 2023-10-20T03:36:19.218 INFO:tasks.rados.rados.0.smithi170.stdout:345: finishing rollback tid 2 to smithi17039300-5 2023-10-20T03:36:19.218 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 5 v 73 (ObjNum 133 snap 27 seq_num 133) dirty exists 2023-10-20T03:36:19.218 INFO:tasks.rados.rados.0.smithi170.stdout:345: done (3 left) 2023-10-20T03:36:19.218 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 20 current snap is 31 2023-10-20T03:36:19.220 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:36:19.219+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:36:19.223 INFO:tasks.rados.rados.0.smithi170.stdout:348: seq_num 143 ranges {4997120=458752,5455872=475136,5931008=655360,6586368=671744,7258112=147456} 2023-10-20T03:36:19.259 INFO:tasks.rados.rados.0.smithi170.stdout:348: writing smithi17039300-20 from 4997120 to 5455872 tid 1 2023-10-20T03:36:19.262 INFO:tasks.rados.rados.0.smithi170.stdout:348: writing smithi17039300-20 from 5455872 to 5931008 tid 2 2023-10-20T03:36:19.268 INFO:tasks.rados.rados.0.smithi170.stdout:348: writing smithi17039300-20 from 5931008 to 6586368 tid 3 2023-10-20T03:36:19.273 INFO:tasks.rados.rados.0.smithi170.stdout:348: writing smithi17039300-20 from 6586368 to 7258112 tid 4 2023-10-20T03:36:19.274 INFO:tasks.rados.rados.0.smithi170.stdout:348: writing smithi17039300-20 from 7258112 to 7405568 tid 5 2023-10-20T03:36:19.275 INFO:tasks.rados.rados.0.smithi170.stdout:349: rollback oid 3 current snap is 31 2023-10-20T03:36:19.275 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 3 to 28 2023-10-20T03:36:19.275 INFO:tasks.rados.rados.0.smithi170.stdout:343: expect (ObjNum 114 snap 19 seq_num 114) 2023-10-20T03:36:19.291 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:19.291 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:36:19.291 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:36:19.291 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:36:19.292 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.292 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:36:19.292 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:36:19.292 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 53.931657321000003, 2023-10-20T03:36:19.292 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010383547999999999, 2023-10-20T03:36:19.292 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.293 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:19.293 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.293 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.293 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.293 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:36:19.294 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.294 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.294 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.294 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.294 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:36:19.294 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.295 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.295 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.295 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356840+0000", 2023-10-20T03:36:19.295 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1999999999999999e-07 2023-10-20T03:36:19.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.296 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.296 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357506+0000", 2023-10-20T03:36:19.296 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00066620500000000001 2023-10-20T03:36:19.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.296 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.297 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.297 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357522+0000", 2023-10-20T03:36:19.297 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5964000000000002e-05 2023-10-20T03:36:19.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.297 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.298 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357535+0000", 2023-10-20T03:36:19.298 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3046e-05 2023-10-20T03:36:19.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.298 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.298 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357625+0000", 2023-10-20T03:36:19.299 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9691999999999994e-05 2023-10-20T03:36:19.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.299 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.299 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:19.299 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357636+0000", 2023-10-20T03:36:19.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1202e-05 2023-10-20T03:36:19.300 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.300 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.300 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:19.300 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.367176+0000", 2023-10-20T03:36:19.300 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0095396060000000008 2023-10-20T03:36:19.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.301 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.301 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.301 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.367223+0000", 2023-10-20T03:36:19.301 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7113000000000001e-05 2023-10-20T03:36:19.301 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.302 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.302 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.302 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.302 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=85, reqid=client.4570.0:312, at_version=43'25, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:36:19.302 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:36:19.303 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 53.914144923000002, 2023-10-20T03:36:19.303 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0052771249999999997, 2023-10-20T03:36:19.303 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.303 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:19.303 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.303 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.304 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:36:19.304 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.304 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.304 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:36:19.305 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.305 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.305 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.305 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374353+0000", 2023-10-20T03:36:19.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6199999999999999e-06 2023-10-20T03:36:19.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.306 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.306 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.306 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374496+0000", 2023-10-20T03:36:19.306 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014282800000000001 2023-10-20T03:36:19.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.307 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.307 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.307 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374515+0000", 2023-10-20T03:36:19.307 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8281e-05 2023-10-20T03:36:19.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.308 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.308 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.308 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374527+0000", 2023-10-20T03:36:19.308 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2835e-05 2023-10-20T03:36:19.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.309 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.309 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374607+0000", 2023-10-20T03:36:19.309 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9283999999999995e-05 2023-10-20T03:36:19.309 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.309 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:19.310 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374617+0000", 2023-10-20T03:36:19.310 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0394000000000001e-05 2023-10-20T03:36:19.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.310 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.310 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:19.311 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.379594+0000", 2023-10-20T03:36:19.311 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049767650000000002 2023-10-20T03:36:19.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.311 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.311 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.312 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.379629+0000", 2023-10-20T03:36:19.312 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5117999999999998e-05 2023-10-20T03:36:19.312 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.312 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.312 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.312 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.313 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.313 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=88, reqid=client.4570.0:313, at_version=43'26, trim_to=0'0, roll_forward_to=43'24))", 2023-10-20T03:36:19.313 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:36:19.313 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 53.900954620999997, 2023-10-20T03:36:19.313 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049208230000000004, 2023-10-20T03:36:19.313 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.314 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:19.314 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.314 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.314 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.314 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:36:19.314 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.315 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.315 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.315 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:36:19.315 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.316 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.316 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387544+0000", 2023-10-20T03:36:19.316 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6109999999999999e-06 2023-10-20T03:36:19.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.316 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.317 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387771+0000", 2023-10-20T03:36:19.317 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022766 2023-10-20T03:36:19.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.317 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.317 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387788+0000", 2023-10-20T03:36:19.318 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7079999999999999e-05 2023-10-20T03:36:19.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.318 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.318 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.318 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387799+0000", 2023-10-20T03:36:19.319 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0608000000000001e-05 2023-10-20T03:36:19.319 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.319 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.319 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.319 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387881+0000", 2023-10-20T03:36:19.319 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2060999999999999e-05 2023-10-20T03:36:19.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.320 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.320 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:19.320 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387892+0000", 2023-10-20T03:36:19.320 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0912e-05 2023-10-20T03:36:19.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.321 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:19.321 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.392420+0000", 2023-10-20T03:36:19.321 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0045283320000000004 2023-10-20T03:36:19.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.321 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.322 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.322 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.392463+0000", 2023-10-20T03:36:19.322 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2558999999999997e-05 2023-10-20T03:36:19.322 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.322 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.322 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.323 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.323 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:333 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~606208] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:19.323 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:36:19.323 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.902566849000003, 2023-10-20T03:36:19.323 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036441440999999998, 2023-10-20T03:36:19.324 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.324 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:19.324 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:19.324 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:19.324 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:19.324 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 333 2023-10-20T03:36:19.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.325 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.325 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.325 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.325 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:36:19.325 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.326 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.326 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.326 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:36:19.326 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.327 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.327 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.327 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385932+0000", 2023-10-20T03:36:19.327 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9929999999999998e-06 2023-10-20T03:36:19.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.328 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.328 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386889+0000", 2023-10-20T03:36:19.328 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000957437 2023-10-20T03:36:19.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.328 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.329 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386904+0000", 2023-10-20T03:36:19.329 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4669e-05 2023-10-20T03:36:19.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.329 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.329 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386910+0000", 2023-10-20T03:36:19.330 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3040000000000001e-06 2023-10-20T03:36:19.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.330 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.330 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.330 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386972+0000", 2023-10-20T03:36:19.331 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2003999999999998e-05 2023-10-20T03:36:19.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.331 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.331 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:19.331 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387043+0000", 2023-10-20T03:36:19.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0591000000000002e-05 2023-10-20T03:36:19.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.332 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.332 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:19.332 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388080+0000", 2023-10-20T03:36:19.332 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010373190000000001 2023-10-20T03:36:19.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.333 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:19.333 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393723+0000", 2023-10-20T03:36:19.333 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056426899999999997 2023-10-20T03:36:19.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.334 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:19.334 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422308+0000", 2023-10-20T03:36:19.334 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028584822999999999 2023-10-20T03:36:19.334 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.334 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.335 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.335 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422371+0000", 2023-10-20T03:36:19.335 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3610999999999998e-05 2023-10-20T03:36:19.335 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.335 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.335 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.336 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.336 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:334 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~720896] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:19.336 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:36:19.336 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.895432450000001, 2023-10-20T03:36:19.336 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029571147999999998, 2023-10-20T03:36:19.337 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.337 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:19.337 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:19.337 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:19.337 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:19.338 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 334 2023-10-20T03:36:19.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.338 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.338 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.338 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.338 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:36:19.339 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.339 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.339 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.339 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:36:19.339 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.340 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.340 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393066+0000", 2023-10-20T03:36:19.340 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8819999999999999e-06 2023-10-20T03:36:19.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.341 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.341 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393827+0000", 2023-10-20T03:36:19.341 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000760844 2023-10-20T03:36:19.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.341 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.342 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.342 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393842+0000", 2023-10-20T03:36:19.342 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5432999999999999e-05 2023-10-20T03:36:19.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.342 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.342 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.343 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393850+0000", 2023-10-20T03:36:19.343 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0199999999999994e-06 2023-10-20T03:36:19.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.343 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.343 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.343 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393915+0000", 2023-10-20T03:36:19.344 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4123999999999998e-05 2023-10-20T03:36:19.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.344 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.344 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:19.344 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394021+0000", 2023-10-20T03:36:19.344 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010668299999999999 2023-10-20T03:36:19.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.345 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.345 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:19.345 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394832+0000", 2023-10-20T03:36:19.345 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081066099999999996 2023-10-20T03:36:19.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.346 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.346 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:19.346 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405535+0000", 2023-10-20T03:36:19.346 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010703351 2023-10-20T03:36:19.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.347 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:19.347 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422553+0000", 2023-10-20T03:36:19.347 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017018021000000001 2023-10-20T03:36:19.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.347 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.348 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.348 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422635+0000", 2023-10-20T03:36:19.348 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2128999999999999e-05 2023-10-20T03:36:19.348 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.348 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.349 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.349 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.349 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.349 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:335 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~49152] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:19.349 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:36:19.350 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.894625474000001, 2023-10-20T03:36:19.350 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029832813999999999, 2023-10-20T03:36:19.350 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.350 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:19.350 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:19.350 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:19.351 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:19.351 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 335 2023-10-20T03:36:19.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.351 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.351 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.351 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.352 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:36:19.352 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.352 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.352 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.352 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.352 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:36:19.353 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.353 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.353 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.353 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393872+0000", 2023-10-20T03:36:19.353 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6000000000000003e-07 2023-10-20T03:36:19.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.354 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.354 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.354 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393919+0000", 2023-10-20T03:36:19.354 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6576000000000003e-05 2023-10-20T03:36:19.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.355 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.355 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.355 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393928+0000", 2023-10-20T03:36:19.355 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1220000000000001e-06 2023-10-20T03:36:19.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.356 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.356 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.356 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393937+0000", 2023-10-20T03:36:19.356 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1740000000000006e-06 2023-10-20T03:36:19.356 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.356 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.357 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.357 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401885+0000", 2023-10-20T03:36:19.357 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079486149999999992 2023-10-20T03:36:19.357 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.357 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.357 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:19.358 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.402012+0000", 2023-10-20T03:36:19.358 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012668 2023-10-20T03:36:19.358 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.358 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.358 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:19.359 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403063+0000", 2023-10-20T03:36:19.359 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010508410000000001 2023-10-20T03:36:19.359 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.359 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.359 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:19.359 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405538+0000", 2023-10-20T03:36:19.360 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024754519999999999 2023-10-20T03:36:19.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.360 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.360 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:19.360 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.423630+0000", 2023-10-20T03:36:19.360 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018091716000000001 2023-10-20T03:36:19.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.361 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.361 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.361 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.423704+0000", 2023-10-20T03:36:19.361 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3878e-05 2023-10-20T03:36:19.361 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.362 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.362 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.362 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.362 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.362 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:336 3.ds0 3:b8ff490e:::smithi17039300-34:head [setxattr _header (56) in=7b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:19.362 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:36:19.363 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.894546099000003, 2023-10-20T03:36:19.363 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030244758, 2023-10-20T03:36:19.363 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.363 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:19.363 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:19.363 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:19.364 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:19.364 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 336 2023-10-20T03:36:19.364 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.364 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.364 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.364 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.365 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:36:19.365 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.365 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.365 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.365 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:36:19.366 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.366 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.366 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.366 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:36:19.366 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8400000000000004e-07 2023-10-20T03:36:19.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.367 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.367 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.367 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393955+0000", 2023-10-20T03:36:19.367 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8210000000000003e-06 2023-10-20T03:36:19.368 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.368 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.368 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.368 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393963+0000", 2023-10-20T03:36:19.368 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8359999999999992e-06 2023-10-20T03:36:19.368 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.368 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.369 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.369 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393968+0000", 2023-10-20T03:36:19.369 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3689999999999998e-06 2023-10-20T03:36:19.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.369 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.370 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.370 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404110+0000", 2023-10-20T03:36:19.370 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010141582 2023-10-20T03:36:19.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.370 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.370 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:19.371 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404246+0000", 2023-10-20T03:36:19.371 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013631799999999999 2023-10-20T03:36:19.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.371 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.371 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:19.371 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405009+0000", 2023-10-20T03:36:19.372 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00076302800000000001 2023-10-20T03:36:19.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.372 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.372 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:19.372 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406198+0000", 2023-10-20T03:36:19.372 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011889610000000001 2023-10-20T03:36:19.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.373 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.373 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:19.373 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424049+0000", 2023-10-20T03:36:19.373 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017850957000000001 2023-10-20T03:36:19.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.374 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.374 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.374 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424195+0000", 2023-10-20T03:36:19.374 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014630200000000001 2023-10-20T03:36:19.374 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.374 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.375 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.375 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.375 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:337 3.ds0 3:b8ff490e:::smithi17039300-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:19.375 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:36:19.375 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 52.894519314999997, 2023-10-20T03:36:19.376 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.032645448000000001, 2023-10-20T03:36:19.376 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.376 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:36:19.376 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:19.376 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:19.376 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:19.377 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 337 2023-10-20T03:36:19.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.377 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.377 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.377 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:36:19.378 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.378 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.378 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.378 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:36:19.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.379 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.379 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.379 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393978+0000", 2023-10-20T03:36:19.379 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0800000000000005e-07 2023-10-20T03:36:19.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.380 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.380 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.380 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393981+0000", 2023-10-20T03:36:19.380 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7199999999999998e-06 2023-10-20T03:36:19.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.381 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.381 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.381 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393988+0000", 2023-10-20T03:36:19.381 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5939999999999996e-06 2023-10-20T03:36:19.381 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.381 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.382 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.382 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393993+0000", 2023-10-20T03:36:19.382 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0200000000000002e-06 2023-10-20T03:36:19.382 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.382 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.382 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.383 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405399+0000", 2023-10-20T03:36:19.383 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011406229 2023-10-20T03:36:19.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.383 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.383 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:19.383 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405522+0000", 2023-10-20T03:36:19.384 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012226199999999999 2023-10-20T03:36:19.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.384 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.384 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.384 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424833+0000", 2023-10-20T03:36:19.384 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019311524 2023-10-20T03:36:19.385 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.385 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.385 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:19.385 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424976+0000", 2023-10-20T03:36:19.385 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014297600000000001 2023-10-20T03:36:19.386 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.386 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.386 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.386 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.426623+0000", 2023-10-20T03:36:19.386 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0016466149999999999 2023-10-20T03:36:19.386 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.387 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.387 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.387 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.387 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:384 3.ds0 3:ba203010:::smithi17039300-4:head [copy-from ver 21] snapc f=[f,d,b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:36:19.387 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:36:19.388 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.917736032999997, 2023-10-20T03:36:19.388 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.077402023, 2023-10-20T03:36:19.388 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.388 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:19.388 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:19.388 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:19.389 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:19.389 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 384 2023-10-20T03:36:19.389 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.389 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.389 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.389 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.390 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:36:19.390 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.390 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.390 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.391 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:36:19.391 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.391 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.391 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.391 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370763+0000", 2023-10-20T03:36:19.392 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4569999999999999e-06 2023-10-20T03:36:19.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.392 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.392 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.392 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370776+0000", 2023-10-20T03:36:19.392 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2483000000000001e-05 2023-10-20T03:36:19.393 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.393 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.393 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.393 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370809+0000", 2023-10-20T03:36:19.393 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2916000000000001e-05 2023-10-20T03:36:19.393 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.394 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.394 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.394 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370826+0000", 2023-10-20T03:36:19.394 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7183000000000001e-05 2023-10-20T03:36:19.394 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.394 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.395 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.395 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371075+0000", 2023-10-20T03:36:19.395 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000249387 2023-10-20T03:36:19.395 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.395 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.395 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:19.396 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371389+0000", 2023-10-20T03:36:19.396 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031429899999999998 2023-10-20T03:36:19.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.396 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.396 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:19.396 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392222+0000", 2023-10-20T03:36:19.397 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020832532000000001 2023-10-20T03:36:19.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.397 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.397 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:19.397 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.447199+0000", 2023-10-20T03:36:19.397 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054976917 2023-10-20T03:36:19.398 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.398 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.398 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:19.398 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.448009+0000", 2023-10-20T03:36:19.398 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080978099999999998 2023-10-20T03:36:19.399 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.399 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.399 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.399 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.448163+0000", 2023-10-20T03:36:19.399 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015406799999999999 2023-10-20T03:36:19.399 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.400 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.400 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.400 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.400 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.400 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:385 3.ds0 3:ba203010:::smithi17039300-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:36:19.400 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:36:19.401 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.917635908999998, 2023-10-20T03:36:19.401 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.074370642000000001, 2023-10-20T03:36:19.401 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.401 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:36:19.401 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:19.401 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:19.402 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:19.402 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 385 2023-10-20T03:36:19.402 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.402 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.402 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.402 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.403 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:36:19.403 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.403 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.403 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.403 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.404 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:36:19.404 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.404 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.404 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.404 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.404 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370862+0000", 2023-10-20T03:36:19.405 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2699999999999998e-07 2023-10-20T03:36:19.405 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.405 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.405 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.405 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370866+0000", 2023-10-20T03:36:19.405 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.42e-06 2023-10-20T03:36:19.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.406 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.406 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.406 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370880+0000", 2023-10-20T03:36:19.406 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3833e-05 2023-10-20T03:36:19.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.407 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.407 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.407 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370889+0000", 2023-10-20T03:36:19.407 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5900000000000008e-06 2023-10-20T03:36:19.407 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.407 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.408 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.408 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371887+0000", 2023-10-20T03:36:19.408 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00099814499999999998 2023-10-20T03:36:19.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.408 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.408 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2023-10-20T03:36:19.409 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.372069+0000", 2023-10-20T03:36:19.409 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018220899999999999 2023-10-20T03:36:19.409 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.409 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.409 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.409 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445044+0000", 2023-10-20T03:36:19.410 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072974686999999996 2023-10-20T03:36:19.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.410 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.410 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:19.410 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445137+0000", 2023-10-20T03:36:19.410 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3694000000000001e-05 2023-10-20T03:36:19.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.411 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.411 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.411 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445231+0000", 2023-10-20T03:36:19.411 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4136999999999997e-05 2023-10-20T03:36:19.411 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.412 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.412 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.412 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.412 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.412 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(osd.12.22:4 3.ds0 3:b8ff490e:::smithi17039300-34:head [assert-version v21,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:36:19.412 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:36:19.413 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.915187185999997, 2023-10-20T03:36:19.413 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083526739999999992, 2023-10-20T03:36:19.413 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.413 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:36:19.413 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.413 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.414 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.414 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:36:19.414 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.414 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.414 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.415 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.415 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:36:19.415 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.415 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.415 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.415 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.416 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373312+0000", 2023-10-20T03:36:19.416 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.424e-06 2023-10-20T03:36:19.416 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.416 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.416 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.416 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373326+0000", 2023-10-20T03:36:19.417 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4226000000000001e-05 2023-10-20T03:36:19.417 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.417 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.417 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.417 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373357+0000", 2023-10-20T03:36:19.417 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0698999999999999e-05 2023-10-20T03:36:19.418 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.418 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.418 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.418 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373388+0000", 2023-10-20T03:36:19.418 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0846000000000002e-05 2023-10-20T03:36:19.418 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.419 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.419 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.419 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373551+0000", 2023-10-20T03:36:19.419 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016350300000000001 2023-10-20T03:36:19.419 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.419 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.420 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:19.420 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373703+0000", 2023-10-20T03:36:19.420 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015216200000000001 2023-10-20T03:36:19.420 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.420 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.420 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.421 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.381662+0000", 2023-10-20T03:36:19.421 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079588139999999998 2023-10-20T03:36:19.421 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.421 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.421 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.421 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.421 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.422 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpReadReply(3.ds0 48/31 ECSubReadReply(tid=59, attrs_read=0))", 2023-10-20T03:36:19.422 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:36:19.422 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.912254668000003, 2023-10-20T03:36:19.422 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0055763890000000002, 2023-10-20T03:36:19.422 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.423 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:19.423 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.423 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.423 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.423 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:36:19.423 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.424 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.424 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.424 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.424 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:36:19.424 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.424 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.425 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.425 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.425 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376244+0000", 2023-10-20T03:36:19.425 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7570000000000001e-06 2023-10-20T03:36:19.425 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.425 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.426 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.426 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380643+0000", 2023-10-20T03:36:19.426 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00439931 2023-10-20T03:36:19.426 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.426 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.426 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.427 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380671+0000", 2023-10-20T03:36:19.427 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8019000000000001e-05 2023-10-20T03:36:19.427 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.427 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.427 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.428 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380685+0000", 2023-10-20T03:36:19.428 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4334e-05 2023-10-20T03:36:19.428 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.428 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.428 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.428 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380780+0000", 2023-10-20T03:36:19.429 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4087999999999998e-05 2023-10-20T03:36:19.429 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.429 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.429 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.429 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.381818+0000", 2023-10-20T03:36:19.429 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001038881 2023-10-20T03:36:19.430 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.430 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.430 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.430 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.430 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.431 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs2 48/47 ECSubWrite(tid=115, reqid=osd.13.0:115, at_version=48'27, trim_to=0'0, roll_forward_to=0'0))", 2023-10-20T03:36:19.431 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:36:19.431 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.898956796, 2023-10-20T03:36:19.431 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047760049999999998, 2023-10-20T03:36:19.431 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.431 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:19.432 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.432 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.432 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.432 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:36:19.432 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.432 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.433 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.433 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.433 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:36:19.433 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.433 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.433 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.434 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.434 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389541+0000", 2023-10-20T03:36:19.434 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5659999999999999e-06 2023-10-20T03:36:19.434 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.434 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.435 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.435 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389579+0000", 2023-10-20T03:36:19.435 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7070999999999998e-05 2023-10-20T03:36:19.435 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.435 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.435 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.436 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389605+0000", 2023-10-20T03:36:19.436 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6848000000000001e-05 2023-10-20T03:36:19.436 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.436 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.436 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.436 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389626+0000", 2023-10-20T03:36:19.437 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0579000000000001e-05 2023-10-20T03:36:19.437 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.437 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.437 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.437 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389781+0000", 2023-10-20T03:36:19.437 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015502300000000001 2023-10-20T03:36:19.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.438 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.438 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:19.438 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389807+0000", 2023-10-20T03:36:19.438 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6418999999999999e-05 2023-10-20T03:36:19.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.439 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.439 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:19.439 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394222+0000", 2023-10-20T03:36:19.439 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044143139999999999 2023-10-20T03:36:19.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.440 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.440 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.440 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394316+0000", 2023-10-20T03:36:19.440 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4184999999999994e-05 2023-10-20T03:36:19.440 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.440 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.441 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.441 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.441 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.441 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:19.441 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:36:19.441 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.859081684000003, 2023-10-20T03:36:19.442 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015918188, 2023-10-20T03:36:19.442 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.442 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:19.442 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.442 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.443 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.443 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:36:19.443 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.443 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.444 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.444 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:36:19.444 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.444 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.444 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.444 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.445 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429417+0000", 2023-10-20T03:36:19.445 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9429999999999999e-06 2023-10-20T03:36:19.445 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.445 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.445 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.446 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429432+0000", 2023-10-20T03:36:19.446 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4942e-05 2023-10-20T03:36:19.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.446 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.446 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.447 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429459+0000", 2023-10-20T03:36:19.447 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6740000000000001e-05 2023-10-20T03:36:19.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.447 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.447 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.447 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429472+0000", 2023-10-20T03:36:19.448 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3382e-05 2023-10-20T03:36:19.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.448 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.448 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.448 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445294+0000", 2023-10-20T03:36:19.449 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015821894999999999 2023-10-20T03:36:19.449 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.449 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.449 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.449 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445333+0000", 2023-10-20T03:36:19.449 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9286000000000003e-05 2023-10-20T03:36:19.450 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.450 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.450 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.450 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.450 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.451 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:19.451 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:36:19.451 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.858732701000001, 2023-10-20T03:36:19.451 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01562377, 2023-10-20T03:36:19.451 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.451 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:19.452 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.452 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.452 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.452 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:36:19.452 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.452 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.453 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.453 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.453 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:36:19.453 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.453 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.454 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.454 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.454 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429765+0000", 2023-10-20T03:36:19.454 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0079999999999999e-06 2023-10-20T03:36:19.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.454 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.455 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.455 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429772+0000", 2023-10-20T03:36:19.455 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2230000000000001e-06 2023-10-20T03:36:19.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.455 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.456 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.456 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429793+0000", 2023-10-20T03:36:19.456 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0707999999999999e-05 2023-10-20T03:36:19.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.456 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.457 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429806+0000", 2023-10-20T03:36:19.457 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3528e-05 2023-10-20T03:36:19.457 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.457 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.457 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.458 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445370+0000", 2023-10-20T03:36:19.458 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015563917 2023-10-20T03:36:19.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.458 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.458 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.459 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445388+0000", 2023-10-20T03:36:19.459 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7385999999999999e-05 2023-10-20T03:36:19.459 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.459 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.459 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.460 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.460 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:19.460 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:36:19.460 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.857081805, 2023-10-20T03:36:19.460 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014045581999999999, 2023-10-20T03:36:19.461 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.461 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:19.461 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.461 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.461 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.462 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:36:19.462 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.462 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.462 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.463 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:36:19.463 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.463 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.464 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431417+0000", 2023-10-20T03:36:19.464 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4159999999999998e-06 2023-10-20T03:36:19.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.464 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.464 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.464 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431433+0000", 2023-10-20T03:36:19.465 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5653000000000001e-05 2023-10-20T03:36:19.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.465 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.465 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.465 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431472+0000", 2023-10-20T03:36:19.466 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9232e-05 2023-10-20T03:36:19.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.466 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.466 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.466 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431492+0000", 2023-10-20T03:36:19.467 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9550000000000001e-05 2023-10-20T03:36:19.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.467 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.467 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.467 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445435+0000", 2023-10-20T03:36:19.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01394307 2023-10-20T03:36:19.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.468 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.468 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445460+0000", 2023-10-20T03:36:19.468 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5661e-05 2023-10-20T03:36:19.469 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.469 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.469 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.469 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.469 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:19.470 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:36:19.470 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.856275805000003, 2023-10-20T03:36:19.470 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013263258999999999, 2023-10-20T03:36:19.470 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.470 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:19.471 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.471 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.471 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.471 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:36:19.471 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.472 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.472 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.472 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:36:19.472 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.473 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.473 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432228+0000", 2023-10-20T03:36:19.473 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.08e-06 2023-10-20T03:36:19.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.473 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.474 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.474 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432236+0000", 2023-10-20T03:36:19.474 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6629999999999993e-06 2023-10-20T03:36:19.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.474 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.475 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432260+0000", 2023-10-20T03:36:19.475 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3998999999999998e-05 2023-10-20T03:36:19.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.475 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432272+0000", 2023-10-20T03:36:19.476 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2023e-05 2023-10-20T03:36:19.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.476 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.476 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.476 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445470+0000", 2023-10-20T03:36:19.477 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013198507 2023-10-20T03:36:19.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.477 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.477 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.477 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445484+0000", 2023-10-20T03:36:19.478 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3987e-05 2023-10-20T03:36:19.478 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.478 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.478 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.478 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.479 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:19.479 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:36:19.479 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 48.855971455999999, 2023-10-20T03:36:19.479 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012997794, 2023-10-20T03:36:19.479 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.479 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:19.480 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.480 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.480 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:36:19.480 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.481 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.481 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:36:19.481 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.481 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.482 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432526+0000", 2023-10-20T03:36:19.482 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1400000000000001e-06 2023-10-20T03:36:19.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.482 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.483 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432532+0000", 2023-10-20T03:36:19.483 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.823e-06 2023-10-20T03:36:19.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.483 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.483 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432552+0000", 2023-10-20T03:36:19.484 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0032999999999999e-05 2023-10-20T03:36:19.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.484 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.484 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.484 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432567+0000", 2023-10-20T03:36:19.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4932999999999999e-05 2023-10-20T03:36:19.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.485 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.485 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.485 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445506+0000", 2023-10-20T03:36:19.485 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012939315 2023-10-20T03:36:19.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.486 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.486 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.486 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445523+0000", 2023-10-20T03:36:19.486 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6549999999999999e-05 2023-10-20T03:36:19.486 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.487 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.487 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.487 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.487 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs2 49/47 ECSubWrite(tid=118, reqid=osd.13.0:118, at_version=49'29, trim_to=0'0, roll_forward_to=48'27))", 2023-10-20T03:36:19.487 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:36:19.488 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 47.877638224999998, 2023-10-20T03:36:19.488 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043378410000000003, 2023-10-20T03:36:19.488 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.488 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:19.488 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.488 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.489 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:36:19.489 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.489 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.490 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:36:19.490 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.490 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410859+0000", 2023-10-20T03:36:19.491 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0699999999999999e-07 2023-10-20T03:36:19.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.491 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.491 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.491 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410884+0000", 2023-10-20T03:36:19.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.529e-05 2023-10-20T03:36:19.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.492 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.492 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.492 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410898+0000", 2023-10-20T03:36:19.492 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.383e-05 2023-10-20T03:36:19.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.493 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.493 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.493 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410928+0000", 2023-10-20T03:36:19.493 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0329e-05 2023-10-20T03:36:19.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.494 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.494 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.411037+0000", 2023-10-20T03:36:19.494 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010869099999999999 2023-10-20T03:36:19.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.495 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.495 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:19.495 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.411048+0000", 2023-10-20T03:36:19.495 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0905e-05 2023-10-20T03:36:19.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.496 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:19.496 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415159+0000", 2023-10-20T03:36:19.496 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041115520000000001 2023-10-20T03:36:19.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.496 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.497 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.497 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415196+0000", 2023-10-20T03:36:19.497 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7036999999999998e-05 2023-10-20T03:36:19.497 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.497 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.497 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.498 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.498 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpRead(3.cs2 55/54 ECSubRead(tid=70, to_read={3:3382ea17:::smithi17039300-35:6=0,262144,0}, subchunks={3:3382ea17:::smithi17039300-35:6=[0,1]}, attrs_to_read=))", 2023-10-20T03:36:19.498 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:37.459240+0000", 2023-10-20T03:36:19.498 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 41.829257118999998, 2023-10-20T03:36:19.498 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044608460000000001, 2023-10-20T03:36:19.499 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:19.499 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:19.499 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:19.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.499 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:19.499 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459240+0000", 2023-10-20T03:36:19.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.500 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.500 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:19.500 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459240+0000", 2023-10-20T03:36:19.500 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:19.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.501 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:19.501 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459241+0000", 2023-10-20T03:36:19.501 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.84e-06 2023-10-20T03:36:19.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.501 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.502 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:19.502 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459256+0000", 2023-10-20T03:36:19.502 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.454e-05 2023-10-20T03:36:19.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.502 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:19.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459288+0000", 2023-10-20T03:36:19.503 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2329000000000002e-05 2023-10-20T03:36:19.503 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.503 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.503 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:19.503 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459307+0000", 2023-10-20T03:36:19.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9046999999999999e-05 2023-10-20T03:36:19.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.504 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.504 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:19.504 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459460+0000", 2023-10-20T03:36:19.504 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015253200000000001 2023-10-20T03:36:19.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:19.505 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:19.505 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:19.505 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.463700+0000", 2023-10-20T03:36:19.505 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042405580000000002 2023-10-20T03:36:19.505 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.506 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.506 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.506 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:19.506 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:19.506 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:19.507 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:36:19.508 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:19.507+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:19.509 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-10-20T03:36:19.510 INFO:tasks.rados.rados.0.smithi170.stdout:344: expect (ObjNum 116 snap 22 seq_num 116) 2023-10-20T03:36:19.511 INFO:tasks.rados.rados.0.smithi170.stdout:347: finishing rollback tid 0 to smithi17039300-8 2023-10-20T03:36:19.511 INFO:tasks.rados.rados.0.smithi170.stdout:347: finishing rollback tid 1 to smithi17039300-8 2023-10-20T03:36:19.511 INFO:tasks.rados.rados.0.smithi170.stdout:347: finishing rollback tid 2 to smithi17039300-8 2023-10-20T03:36:19.511 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 8 v 83 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-10-20T03:36:19.511 INFO:tasks.rados.rados.0.smithi170.stdout:349: finishing rollback tid 0 to smithi17039300-3 2023-10-20T03:36:19.511 INFO:tasks.rados.rados.0.smithi170.stdout:348: finishing write tid 1 to smithi17039300-20 2023-10-20T03:36:19.512 INFO:tasks.rados.rados.0.smithi170.stdout:349: finishing rollback tid 1 to smithi17039300-3 2023-10-20T03:36:19.512 INFO:tasks.rados.rados.0.smithi170.stdout:349: finishing rollback tid 2 to smithi17039300-3 2023-10-20T03:36:19.512 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 3 v 86 (ObjNum 2 snap 0 seq_num 2) dirty exists 2023-10-20T03:36:19.512 INFO:tasks.rados.rados.0.smithi170.stdout:348: finishing write tid 2 to smithi17039300-20 2023-10-20T03:36:19.512 INFO:tasks.rados.rados.0.smithi170.stdout:343: done (4 left) 2023-10-20T03:36:19.512 INFO:tasks.rados.rados.0.smithi170.stdout:344: done (3 left) 2023-10-20T03:36:19.513 INFO:tasks.rados.rados.0.smithi170.stdout:347: done (2 left) 2023-10-20T03:36:19.513 INFO:tasks.rados.rados.0.smithi170.stdout:349: done (1 left) 2023-10-20T03:36:19.513 INFO:tasks.rados.rados.0.smithi170.stdout:350: read oid 28 snap -1 2023-10-20T03:36:19.513 INFO:tasks.rados.rados.0.smithi170.stdout:350: expect (ObjNum 94 snap 13 seq_num 94) 2023-10-20T03:36:19.513 INFO:tasks.rados.rados.0.smithi170.stdout:351: rmattr oid 19 current snap is 31 2023-10-20T03:36:19.513 INFO:tasks.rados.rados.0.smithi170.stdout:351: done (2 left) 2023-10-20T03:36:19.514 INFO:tasks.rados.rados.0.smithi170.stdout:352: rollback oid 25 current snap is 31 2023-10-20T03:36:19.514 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 25 to 28 2023-10-20T03:36:19.514 INFO:tasks.rados.rados.0.smithi170.stdout:348: finishing write tid 3 to smithi17039300-20 2023-10-20T03:36:19.514 INFO:tasks.rados.rados.0.smithi170.stdout:348: finishing write tid 4 to smithi17039300-20 2023-10-20T03:36:19.514 INFO:tasks.rados.rados.0.smithi170.stdout:348: finishing write tid 5 to smithi17039300-20 2023-10-20T03:36:19.514 INFO:tasks.rados.rados.0.smithi170.stdout:348: finishing write tid 6 to smithi17039300-20 2023-10-20T03:36:19.515 INFO:tasks.rados.rados.0.smithi170.stdout:348: finishing write tid 7 to smithi17039300-20 2023-10-20T03:36:19.515 INFO:tasks.rados.rados.0.smithi170.stdout:348: oid 20 updating version 0 to 56 2023-10-20T03:36:19.515 INFO:tasks.rados.rados.0.smithi170.stdout:348: oid 20 version 56 is already newer than 55 2023-10-20T03:36:19.515 INFO:tasks.rados.rados.0.smithi170.stdout:348: oid 20 version 56 is already newer than 54 2023-10-20T03:36:19.515 INFO:tasks.rados.rados.0.smithi170.stdout:348: oid 20 updating version 56 to 58 2023-10-20T03:36:19.516 INFO:tasks.rados.rados.0.smithi170.stdout:348: oid 20 version 58 is already newer than 57 2023-10-20T03:36:19.516 INFO:tasks.rados.rados.0.smithi170.stdout:348: oid 20 updating version 58 to 59 2023-10-20T03:36:19.516 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 20 v 59 (ObjNum 143 snap 31 seq_num 143) dirty exists 2023-10-20T03:36:19.516 INFO:tasks.rados.rados.0.smithi170.stdout:348: left oid 20 (ObjNum 143 snap 31 seq_num 143) 2023-10-20T03:36:19.516 INFO:tasks.rados.rados.0.smithi170.stdout:348: done (2 left) 2023-10-20T03:36:19.516 INFO:tasks.rados.rados.0.smithi170.stdout:353: rollback oid 45 current snap is 31 2023-10-20T03:36:19.517 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 45 to 28 2023-10-20T03:36:19.517 INFO:tasks.rados.rados.0.smithi170.stdout:354: snap_create 2023-10-20T03:36:19.517 INFO:tasks.rados.rados.0.smithi170.stdout:353: finishing rollback tid 0 to smithi17039300-45 2023-10-20T03:36:19.517 INFO:tasks.rados.rados.0.smithi170.stdout:353: finishing rollback tid 1 to smithi17039300-45 2023-10-20T03:36:19.517 INFO:tasks.rados.rados.0.smithi170.stdout:352: finishing rollback tid 0 to smithi17039300-25 2023-10-20T03:36:19.517 INFO:tasks.rados.rados.0.smithi170.stdout:353: finishing rollback tid 2 to smithi17039300-45 2023-10-20T03:36:19.517 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 45 v 29 (ObjNum 44 snap 0 seq_num 44) dirty exists 2023-10-20T03:36:19.518 INFO:tasks.rados.rados.0.smithi170.stdout:350: expect (ObjNum 94 snap 13 seq_num 94) 2023-10-20T03:36:19.518 INFO:tasks.rados.rados.0.smithi170.stdout:352: finishing rollback tid 1 to smithi17039300-25 2023-10-20T03:36:19.518 INFO:tasks.rados.rados.0.smithi170.stdout:352: finishing rollback tid 2 to smithi17039300-25 2023-10-20T03:36:19.518 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 51 (ObjNum 100 snap 13 seq_num 100) dirty exists 2023-10-20T03:36:19.608 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:19.607+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:19.653 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:19.653 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:36:19.653 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:36:19.654 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:19.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-10-20T03:36:19.710 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:36:19.707+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:36:19.711 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:36:19.711 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:36:19.729 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:36:19.729 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:36:19.751 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:36:19.752 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:36:19.774 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:36:19.775 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:36:19.795 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:36:19.795 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:36:19.797 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:19.797 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:19.811 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:19.808+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:19.812 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-10-20T03:36:19.814 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:36:19.814 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:36:19.837 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2023-10-20T03:36:19.837 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-10-20T03:36:19.859 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:36:19.860 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:36:19.879 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:36:19.880 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:36:19.880 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.9 2023-10-20T03:36:19.904 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:36:19.905 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:36:19.911 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:36:19.907+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:36:19.929 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:36:19.929 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:36:19.931 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:19.931 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:19.946 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-10-20T03:36:19.952 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:36:19.952 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:36:19.972 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:36:19.972 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:36:19.996 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:36:19.996 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-10-20T03:36:20.011 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:20.008+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:20.019 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:36:20.020 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:36:20.024 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:36:20.024 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:36:20.075 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:20.075 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:20.089 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-10-20T03:36:20.112 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:20.108+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:20.206 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-10-20T03:36:20.206 INFO:tasks.rados.rados.0.smithi170.stdout:350: done (3 left) 2023-10-20T03:36:20.207 INFO:tasks.rados.rados.0.smithi170.stdout:352: done (2 left) 2023-10-20T03:36:20.207 INFO:tasks.rados.rados.0.smithi170.stdout:353: done (1 left) 2023-10-20T03:36:20.207 INFO:tasks.rados.rados.0.smithi170.stdout:354: done (0 left) 2023-10-20T03:36:20.207 INFO:tasks.rados.rados.0.smithi170.stdout:355: setattr oid 37 current snap is 32 2023-10-20T03:36:20.209 INFO:tasks.rados.rados.0.smithi170.stdout:356: rmattr oid 7 current snap is 32 2023-10-20T03:36:20.209 INFO:tasks.rados.rados.0.smithi170.stdout:356: done (1 left) 2023-10-20T03:36:20.209 INFO:tasks.rados.rados.0.smithi170.stdout:357: read oid 28 snap -1 2023-10-20T03:36:20.210 INFO:tasks.rados.rados.0.smithi170.stdout:357: expect (ObjNum 94 snap 13 seq_num 94) 2023-10-20T03:36:20.210 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 7 current snap is 32 2023-10-20T03:36:20.210 INFO:tasks.rados.rados.0.smithi170.stdout:358: seq_num 144 ranges {1114112=16384} 2023-10-20T03:36:20.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~0s 2023-10-20T03:36:20.215 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:20.215 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:20.218 INFO:tasks.rados.rados.0.smithi170.stdout:358: writing smithi17039300-7 from 1114112 to 1130496 tid 1 2023-10-20T03:36:20.218 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 32 current snap is 32 2023-10-20T03:36:20.218 INFO:tasks.rados.rados.0.smithi170.stdout:359: seq_num 145 ranges {0=32768} 2023-10-20T03:36:20.218 INFO:tasks.rados.rados.0.smithi170.stdout:359: writing smithi17039300-32 from 0 to 32768 tid 1 2023-10-20T03:36:20.218 INFO:tasks.rados.rados.0.smithi170.stdout:360: rollback oid 18 current snap is 32 2023-10-20T03:36:20.219 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 18 to 28 2023-10-20T03:36:20.219 INFO:tasks.rados.rados.0.smithi170.stdout:361: delete oid 50 current snap is 32 2023-10-20T03:36:20.230 INFO:tasks.rados.rados.0.smithi170.stdout:357: expect (ObjNum 94 snap 13 seq_num 94) 2023-10-20T03:36:20.237 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-10-20T03:36:20.254 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:20.252+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:20.292 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 37 v 85 (ObjNum 85 snap 9 seq_num 85) dirty exists 2023-10-20T03:36:20.292 INFO:tasks.rados.rados.0.smithi170.stdout:359: finishing write tid 1 to smithi17039300-32 2023-10-20T03:36:20.292 INFO:tasks.rados.rados.0.smithi170.stdout:360: finishing rollback tid 1 to smithi17039300-18 2023-10-20T03:36:20.293 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 18 v 0 (ObjNum 33 snap 0 seq_num 3157945024) dirty dne 2023-10-20T03:36:20.293 INFO:tasks.rados.rados.0.smithi170.stdout:358: finishing write tid 1 to smithi17039300-7 2023-10-20T03:36:20.293 INFO:tasks.rados.rados.0.smithi170.stdout:355: done (5 left) 2023-10-20T03:36:20.293 INFO:tasks.rados.rados.0.smithi170.stdout:357: done (4 left) 2023-10-20T03:36:20.293 INFO:tasks.rados.rados.0.smithi170.stdout:360: done (3 left) 2023-10-20T03:36:20.293 INFO:tasks.rados.rados.0.smithi170.stdout:361: done (2 left) 2023-10-20T03:36:20.294 INFO:tasks.rados.rados.0.smithi170.stdout:362: delete oid 35 current snap is 32 2023-10-20T03:36:20.294 INFO:tasks.rados.rados.0.smithi170.stdout:359: finishing write tid 2 to smithi17039300-32 2023-10-20T03:36:20.294 INFO:tasks.rados.rados.0.smithi170.stdout:358: finishing write tid 2 to smithi17039300-7 2023-10-20T03:36:20.294 INFO:tasks.rados.rados.0.smithi170.stdout:358: finishing write tid 3 to smithi17039300-7 2023-10-20T03:36:20.294 INFO:tasks.rados.rados.0.smithi170.stdout:358: oid 7 updating version 0 to 75 2023-10-20T03:36:20.295 INFO:tasks.rados.rados.0.smithi170.stdout:358: oid 7 updating version 75 to 76 2023-10-20T03:36:20.295 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 7 v 76 (ObjNum 144 snap 32 seq_num 144) dirty exists 2023-10-20T03:36:20.295 INFO:tasks.rados.rados.0.smithi170.stdout:358: left oid 7 (ObjNum 144 snap 32 seq_num 144) 2023-10-20T03:36:20.295 INFO:tasks.rados.rados.0.smithi170.stdout:359: finishing write tid 3 to smithi17039300-32 2023-10-20T03:36:20.295 INFO:tasks.rados.rados.0.smithi170.stdout:359: oid 32 updating version 0 to 30 2023-10-20T03:36:20.295 INFO:tasks.rados.rados.0.smithi170.stdout:359: oid 32 updating version 30 to 31 2023-10-20T03:36:20.296 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 32 v 31 (ObjNum 145 snap 32 seq_num 145) dirty exists 2023-10-20T03:36:20.296 INFO:tasks.rados.rados.0.smithi170.stdout:359: left oid 32 (ObjNum 145 snap 32 seq_num 145) 2023-10-20T03:36:20.299 INFO:tasks.rados.rados.0.smithi170.stdout:358: done (2 left) 2023-10-20T03:36:20.299 INFO:tasks.rados.rados.0.smithi170.stdout:359: done (1 left) 2023-10-20T03:36:20.299 INFO:tasks.rados.rados.0.smithi170.stdout:362: done (0 left) 2023-10-20T03:36:20.299 INFO:tasks.rados.rados.0.smithi170.stdout:363: setattr oid 20 current snap is 32 2023-10-20T03:36:20.302 INFO:tasks.rados.rados.0.smithi170.stdout:364: snap_remove snap 31 2023-10-20T03:36:20.314 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:36:20.311+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:36:20.364 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:20.364 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:20.380 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-10-20T03:36:20.415 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:20.412+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:20.489 INFO:teuthology.orchestra.run.smithi040.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-10-20T03:36:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-20T03:36:20.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-10-20T03:36:20.516 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:20.512+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:20.602 INFO:teuthology.orchestra.run.smithi040.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-10-20T03:36:20.605 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-20T03:36:20.606 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-10-20T03:36:20.617 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:20.615+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:20.715 INFO:teuthology.orchestra.run.smithi040.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-10-20T03:36:20.718 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-20T03:36:20.719 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-10-20T03:36:20.719 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:20.716+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:20.819 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:20.816+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:20.847 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:20.847 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:20.861 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-10-20T03:36:20.920 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:20.916+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:20.988 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:20.989 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:21.003 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-10-20T03:36:21.051 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:21.048+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:21.064 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15] out_osds: [12] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 10, 14, 3, 7, 11, 15, 6] 2023-10-20T03:36:21.065 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-10-20T03:36:21.065 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2023-10-20T03:36:21.065 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T03:36:21.122 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:21.116+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:21.129 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:21.129 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:21.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-10-20T03:36:21.209 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2023-10-20T03:36:21.216 INFO:tasks.rados.rados.0.smithi170.stdout:364: done (1 left) 2023-10-20T03:36:21.216 INFO:tasks.rados.rados.0.smithi170.stdout:365: snap_remove snap 28 2023-10-20T03:36:21.223 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:21.220+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:21.285 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:21.286 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:21.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-10-20T03:36:21.323 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:36:21.320+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:36:21.405 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:36:21.406 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-10-20T03:36:21.422 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:21.423 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:21.424 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2023-10-20T03:36:21.212061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":104137248,"num_objects":64,"num_object_clones":25,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":6,"num_read":521,"num_read_kb":220969,"num_write":1083,"num_write_kb":136212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":91504640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1111,"ondisk_log_size":1111,"up":127,"acting":127,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":616620,"kb_used_data":184972,"kb_used_omap":28,"kb_used_meta":431395,"kb_avail":1414960980,"statfs":{"total":1449551462400,"available":1448920043520,"internally_reserved":0,"allocated":189411328,"data_stored":237244185,"data_compressed":2255351,"data_compressed_allocated":4669440,"data_compressed_original":10301440,"omap_allocated":29064,"internal_metadata":441749112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9863168,"num_objects":12,"num_object_clones":12,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":5,"num_read":20,"num_read_kb":24159,"num_write":95,"num_write_kb":6912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":10813440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.044568"},"pg_stats":[{"pgid":"3.f","version":"98'85","reported_seq":362,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-20T03:36:20.214836+0000","last_change":"2023-10-20T03:36:19.215280+0000","last_active":"2023-10-20T03:36:20.214836+0000","last_peered":"2023-10-20T03:36:20.214836+0000","last_clean":"2023-10-20T03:36:20.214836+0000","last_became_active":"2023-10-20T03:35:30.382595+0000","last_became_peered":"2023-10-20T03:35:30.382595+0000","last_unstale":"2023-10-20T03:36:20.214836+0000","last_undegraded":"2023-10-20T03:36:20.214836+0000","last_fullsized":"2023-10-20T03:36:20.214836+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'78","last_scrub_stamp":"2023-10-20T03:36:19.084653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:36:19.084653+0000","objects_scrubbed":4,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:37:26.062701+0000","scrub_duration":0.012971172,"objects_trimmed":0,"snaptrim_duration":0.018768579000000001,"stat_sum":{"num_bytes":4603904,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":88,"num_write_kb":7232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,15,14,2,11,1],"acting":[0,3,15,14,2,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"2"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"3.e","version":"98'76","reported_seq":403,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-20T03:36:20.238887+0000","last_change":"2023-10-20T03:36:19.218337+0000","last_active":"2023-10-20T03:36:20.238887+0000","last_peered":"2023-10-20T03:36:20.238887+0000","last_clean":"2023-10-20T03:36:20.238887+0000","last_became_active":"2023-10-20T03:36:15.161134+0000","last_became_peered":"2023-10-20T03:36:15.161134+0000","last_unstale":"2023-10-20T03:36:20.238887+0000","last_undegraded":"2023-10-20T03:36:20.238887+0000","last_fullsized":"2023-10-20T03:36:20.238887+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:35.156996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029686634,"stat_sum":{"num_bytes":12009472,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":47,"num_read_kb":17754,"num_write":107,"num_write_kb":14144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":14647296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,6,4,13,3,5],"acting":[11,14,6,4,13,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"2"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"3.d","version":"97'85","reported_seq":363,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-20T03:36:20.222593+0000","last_change":"2023-10-20T03:36:19.231544+0000","last_active":"2023-10-20T03:36:20.222593+0000","last_peered":"2023-10-20T03:36:20.222593+0000","last_clean":"2023-10-20T03:36:20.222593+0000","last_became_active":"2023-10-20T03:35:38.820900+0000","last_became_peered":"2023-10-20T03:35:38.820900+0000","last_unstale":"2023-10-20T03:36:20.222593+0000","last_undegraded":"2023-10-20T03:36:20.222593+0000","last_fullsized":"2023-10-20T03:36:20.222593+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:24.668541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.022909571,"stat_sum":{"num_bytes":6225920,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":39,"num_read_kb":29247,"num_write":57,"num_write_kb":12896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,11,0,8,13,5],"acting":[1,4,11,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"2"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"3.c","version":"98'66","reported_seq":349,"reported_epoch":98,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:36:20.215334+0000","last_change":"2023-10-20T03:36:17.195721+0000","last_active":"2023-10-20T03:36:20.215334+0000","last_peered":"2023-10-20T03:36:20.215334+0000","last_clean":"2023-10-20T03:36:15.184005+0000","last_became_active":"2023-10-20T03:36:17.195721+0000","last_became_peered":"2023-10-20T03:36:17.195721+0000","last_unstale":"2023-10-20T03:36:20.215334+0000","last_undegraded":"2023-10-20T03:36:17.177610+0000","last_fullsized":"2023-10-20T03:36:17.176354+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012586561,"stat_sum":{"num_bytes":2654208,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":21,"num_read_kb":17432,"num_write":25,"num_write_kb":5760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":573440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,5,3,7,1,2147483647],"acting":[0,13,5,3,7,1,2147483647],"avail_no_missing":["13(1)","0(0)","1(5)","3(3)","5(2)","7(4)"],"object_location_counts":[{"shards":"0(0),1(5),3(3),5(2),7(4),13(1)","objects":2}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"1"}]},{"pgid":"3.b","version":"97'51","reported_seq":345,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-20T03:36:20.214537+0000","last_change":"2023-10-20T03:36:19.190444+0000","last_active":"2023-10-20T03:36:20.214537+0000","last_peered":"2023-10-20T03:36:20.214537+0000","last_clean":"2023-10-20T03:36:20.214537+0000","last_became_active":"2023-10-20T03:35:38.816060+0000","last_became_peered":"2023-10-20T03:35:38.816060+0000","last_unstale":"2023-10-20T03:36:20.214537+0000","last_undegraded":"2023-10-20T03:36:20.214537+0000","last_fullsized":"2023-10-20T03:36:20.214537+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:39.470206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019638289999999998,"stat_sum":{"num_bytes":9502720,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":7388,"num_write":29,"num_write_kb":9680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2752512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,10,0,11,4,14],"acting":[13,1,10,0,11,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"2"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"3.a","version":"98'61","reported_seq":260,"reported_epoch":98,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:36:20.314378+0000","last_change":"2023-10-20T03:36:17.187291+0000","last_active":"2023-10-20T03:36:20.314378+0000","last_peered":"2023-10-20T03:36:20.314378+0000","last_clean":"2023-10-20T03:36:15.260438+0000","last_became_active":"2023-10-20T03:36:17.187291+0000","last_became_peered":"2023-10-20T03:36:17.187291+0000","last_unstale":"2023-10-20T03:36:20.314378+0000","last_undegraded":"2023-10-20T03:36:17.170831+0000","last_fullsized":"2023-10-20T03:36:17.170510+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":30,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:30.529848+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":11,"num_object_clones":7,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":25,"num_read_kb":11184,"num_write":105,"num_write_kb":13104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":10813440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,4,6,10,3,15,8],"acting":[2147483647,4,6,10,3,15,8],"avail_no_missing":["4(1)","3(4)","6(2)","8(6)","10(3)","15(5)"],"object_location_counts":[{"shards":"3(4),4(1),6(2),8(6),10(3),15(5)","objects":11}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"98'55","reported_seq":317,"reported_epoch":98,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:36:20.226846+0000","last_change":"2023-10-20T03:36:17.186951+0000","last_active":"2023-10-20T03:36:20.226846+0000","last_peered":"2023-10-20T03:36:20.226846+0000","last_clean":"2023-10-20T03:36:15.143467+0000","last_became_active":"2023-10-20T03:36:17.186951+0000","last_became_peered":"2023-10-20T03:36:17.186951+0000","last_unstale":"2023-10-20T03:36:20.226846+0000","last_undegraded":"2023-10-20T03:36:17.175240+0000","last_fullsized":"2023-10-20T03:36:17.174150+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:33.305828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010593429999999999,"stat_sum":{"num_bytes":3801088,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":6490,"num_write":29,"num_write_kb":8112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,10,11,4,2147483647,3],"acting":[7,2,10,11,4,2147483647,3],"avail_no_missing":["7(0)","2(1)","3(6)","4(4)","10(2)","11(3)"],"object_location_counts":[{"shards":"2(1),3(6),4(4),7(0),10(2),11(3)","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"1"}]},{"pgid":"3.8","version":"97'29","reported_seq":306,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-20T03:36:19.418182+0000","last_change":"2023-10-20T03:36:19.224617+0000","last_active":"2023-10-20T03:36:19.418182+0000","last_peered":"2023-10-20T03:36:19.418182+0000","last_clean":"2023-10-20T03:36:19.418182+0000","last_became_active":"2023-10-20T03:36:15.158182+0000","last_became_peered":"2023-10-20T03:36:15.158182+0000","last_unstale":"2023-10-20T03:36:19.418182+0000","last_undegraded":"2023-10-20T03:36:19.418182+0000","last_fullsized":"2023-10-20T03:36:19.418182+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'23","last_scrub_stamp":"2023-10-20T03:36:18.357834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:36:18.357834+0000","objects_scrubbed":3,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:37:19.535928+0000","scrub_duration":0.010344032,"objects_trimmed":0,"snaptrim_duration":0.021709118999999999,"stat_sum":{"num_bytes":6307840,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":15,"num_read_kb":12102,"num_write":19,"num_write_kb":6160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11,3,5,0,7],"acting":[6,1,11,3,5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"2"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":234,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-20T03:36:19.325161+0000","last_change":"2023-10-20T03:36:19.325301+0000","last_active":"2023-10-20T03:36:19.325161+0000","last_peered":"2023-10-20T03:36:19.325161+0000","last_clean":"2023-10-20T03:36:19.325161+0000","last_became_active":"2023-10-20T03:36:12.126368+0000","last_became_peered":"2023-10-20T03:36:12.126368+0000","last_unstale":"2023-10-20T03:36:19.325161+0000","last_undegraded":"2023-10-20T03:36:19.325161+0000","last_fullsized":"2023-10-20T03:36:19.325161+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2023-10-20T03:36:19.324962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_clean_scrub_stamp":"2023-10-20T03:36:19.324962+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:37:23.639536+0000","scrub_duration":0.0048410670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":81,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-20T03:36:19.210149+0000","last_change":"2023-10-20T03:35:05.117915+0000","last_active":"2023-10-20T03:36:19.210149+0000","last_peered":"2023-10-20T03:36:19.210149+0000","last_clean":"2023-10-20T03:36:19.210149+0000","last_became_active":"2023-10-20T03:35:05.117258+0000","last_became_peered":"2023-10-20T03:35:05.117258+0000","last_unstale":"2023-10-20T03:36:19.210149+0000","last_undegraded":"2023-10-20T03:36:19.210149+0000","last_fullsized":"2023-10-20T03:36:19.210149+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:32.359335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"97'92","reported_seq":365,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-20T03:36:20.222136+0000","last_change":"2023-10-20T03:36:19.240397+0000","last_active":"2023-10-20T03:36:20.222136+0000","last_peered":"2023-10-20T03:36:20.222136+0000","last_clean":"2023-10-20T03:36:20.222136+0000","last_became_active":"2023-10-20T03:35:14.221482+0000","last_became_peered":"2023-10-20T03:35:14.221482+0000","last_unstale":"2023-10-20T03:36:20.222136+0000","last_undegraded":"2023-10-20T03:36:20.222136+0000","last_fullsized":"2023-10-20T03:36:20.222136+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:31.173549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029658733999999999,"stat_sum":{"num_bytes":7454720,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":17613,"num_write":122,"num_write_kb":9104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,0,4,8,10,7],"acting":[14,3,0,4,8,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"2"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-20T03:36:19.196307+0000","last_change":"2023-10-20T03:35:05.201208+0000","last_active":"2023-10-20T03:36:19.196307+0000","last_peered":"2023-10-20T03:36:19.196307+0000","last_clean":"2023-10-20T03:36:19.196307+0000","last_became_active":"2023-10-20T03:35:05.200512+0000","last_became_peered":"2023-10-20T03:35:05.200512+0000","last_unstale":"2023-10-20T03:36:19.196307+0000","last_undegraded":"2023-10-20T03:36:19.196307+0000","last_fullsized":"2023-10-20T03:36:19.196307+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:30.668122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"97'83","reported_seq":351,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-20T03:36:20.225423+0000","last_change":"2023-10-20T03:36:19.214150+0000","last_active":"2023-10-20T03:36:20.225423+0000","last_peered":"2023-10-20T03:36:20.225423+0000","last_clean":"2023-10-20T03:36:20.225423+0000","last_became_active":"2023-10-20T03:35:14.311165+0000","last_became_peered":"2023-10-20T03:35:14.311165+0000","last_unstale":"2023-10-20T03:36:20.225423+0000","last_undegraded":"2023-10-20T03:36:20.225423+0000","last_fullsized":"2023-10-20T03:36:20.225423+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025708580000000002,"stat_sum":{"num_bytes":7127040,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":13354,"num_write":28,"num_write_kb":7328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,14,10,7,3,2],"acting":[13,4,14,10,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"2"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":81,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-20T03:36:19.209824+0000","last_change":"2023-10-20T03:35:05.113211+0000","last_active":"2023-10-20T03:36:19.209824+0000","last_peered":"2023-10-20T03:36:19.209824+0000","last_clean":"2023-10-20T03:36:19.209824+0000","last_became_active":"2023-10-20T03:35:05.112592+0000","last_became_peered":"2023-10-20T03:35:05.112592+0000","last_unstale":"2023-10-20T03:36:19.209824+0000","last_undegraded":"2023-10-20T03:36:19.209824+0000","last_fullsized":"2023-10-20T03:36:19.209824+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:27.383114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"93'61","reported_seq":368,"reported_epoch":98,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:36:20.214402+0000","last_change":"2023-10-20T03:36:17.189168+0000","last_active":"2023-10-20T03:36:20.214402+0000","last_peered":"2023-10-20T03:36:20.214402+0000","last_clean":"2023-10-20T03:36:15.290892+0000","last_became_active":"2023-10-20T03:36:17.189168+0000","last_became_peered":"2023-10-20T03:36:17.189168+0000","last_unstale":"2023-10-20T03:36:20.214402+0000","last_undegraded":"2023-10-20T03:36:17.176423+0000","last_fullsized":"2023-10-20T03:36:17.172773+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:20.867158+0000","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.13851994500000001,"stat_sum":{"num_bytes":5881856,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":9310,"num_write":52,"num_write_kb":11488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":17907712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4,2,6,15,2147483647],"acting":[3,7,4,2,6,15,2147483647],"avail_no_missing":["3(0)","2(3)","4(2)","6(4)","7(1)","15(5)"],"object_location_counts":[{"shards":"2(3),3(0),4(2),6(4),7(1),15(5)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":84,"reported_epoch":97,"state":"active+undersized","last_fresh":"2023-10-20T03:36:19.196441+0000","last_change":"2023-10-20T03:36:17.172853+0000","last_active":"2023-10-20T03:36:19.196441+0000","last_peered":"2023-10-20T03:36:19.196441+0000","last_clean":"2023-10-20T03:36:15.147401+0000","last_became_active":"2023-10-20T03:36:17.172853+0000","last_became_peered":"2023-10-20T03:36:17.172853+0000","last_unstale":"2023-10-20T03:36:19.196441+0000","last_undegraded":"2023-10-20T03:36:19.196441+0000","last_fullsized":"2023-10-20T03:36:17.171442+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8],"acting":[8],"avail_no_missing":["8"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"98'73","reported_seq":380,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-20T03:36:20.208099+0000","last_change":"2023-10-20T03:36:19.224251+0000","last_active":"2023-10-20T03:36:20.208099+0000","last_peered":"2023-10-20T03:36:20.208099+0000","last_clean":"2023-10-20T03:36:20.208099+0000","last_became_active":"2023-10-20T03:35:38.811204+0000","last_became_peered":"2023-10-20T03:35:38.811204+0000","last_unstale":"2023-10-20T03:36:20.208099+0000","last_undegraded":"2023-10-20T03:36:20.208099+0000","last_fullsized":"2023-10-20T03:36:20.208099+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:29.013371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011832476999999999,"stat_sum":{"num_bytes":12910592,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":31,"num_read_kb":37742,"num_write":41,"num_write_kb":13280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4866048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,1,0,11,2,14],"acting":[10,8,1,0,11,2,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"2"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"3.5","version":"97'32","reported_seq":297,"reported_epoch":98,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:36:20.225571+0000","last_change":"2023-10-20T03:36:17.185985+0000","last_active":"2023-10-20T03:36:20.225571+0000","last_peered":"2023-10-20T03:36:20.225571+0000","last_clean":"2023-10-20T03:36:15.147049+0000","last_became_active":"2023-10-20T03:36:17.185985+0000","last_became_peered":"2023-10-20T03:36:17.185985+0000","last_unstale":"2023-10-20T03:36:20.225571+0000","last_undegraded":"2023-10-20T03:36:17.171750+0000","last_fullsized":"2023-10-20T03:36:17.171338+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:42.159609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085077720000000003,"stat_sum":{"num_bytes":802816,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":16,"num_read_kb":3604,"num_write":30,"num_write_kb":1664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,2,2147483647,1,11],"acting":[8,13,5,2,2147483647,1,11],"avail_no_missing":["8(0)","1(5)","2(3)","5(2)","11(6)","13(1)"],"object_location_counts":[{"shards":"1(5),2(3),5(2),8(0),11(6),13(1)","objects":2}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":81,"reported_epoch":97,"state":"active+undersized","last_fresh":"2023-10-20T03:36:19.206766+0000","last_change":"2023-10-20T03:36:17.187799+0000","last_active":"2023-10-20T03:36:19.206766+0000","last_peered":"2023-10-20T03:36:19.206766+0000","last_clean":"2023-10-20T03:36:15.148140+0000","last_became_active":"2023-10-20T03:36:17.187799+0000","last_became_peered":"2023-10-20T03:36:17.187799+0000","last_unstale":"2023-10-20T03:36:19.206766+0000","last_undegraded":"2023-10-20T03:36:19.206766+0000","last_fullsized":"2023-10-20T03:36:17.182506+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"93'6","reported_seq":281,"reported_epoch":98,"state":"active+undersized","last_fresh":"2023-10-20T03:36:20.223247+0000","last_change":"2023-10-20T03:36:17.192050+0000","last_active":"2023-10-20T03:36:20.223247+0000","last_peered":"2023-10-20T03:36:20.223247+0000","last_clean":"2023-10-20T03:36:15.253957+0000","last_became_active":"2023-10-20T03:36:17.192050+0000","last_became_peered":"2023-10-20T03:36:17.192050+0000","last_unstale":"2023-10-20T03:36:20.223247+0000","last_undegraded":"2023-10-20T03:36:20.223247+0000","last_fullsized":"2023-10-20T03:36:17.181362+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:42.296283+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.091105647999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":290,"num_write":3,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,15,2147483647,2,6,10,13],"acting":[1,15,2147483647,2,6,10,13],"avail_no_missing":["1(0)","2(3)","6(4)","10(5)","13(6)","15(1)"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-20T03:36:19.185480+0000","last_change":"2023-10-20T03:35:05.122638+0000","last_active":"2023-10-20T03:36:19.185480+0000","last_peered":"2023-10-20T03:36:19.185480+0000","last_clean":"2023-10-20T03:36:19.185480+0000","last_became_active":"2023-10-20T03:35:05.122056+0000","last_became_peered":"2023-10-20T03:35:05.122056+0000","last_unstale":"2023-10-20T03:36:19.185480+0000","last_undegraded":"2023-10-20T03:36:19.185480+0000","last_fullsized":"2023-10-20T03:36:19.185480+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:21.001893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"93'13","reported_seq":301,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-20T03:36:20.215015+0000","last_change":"2023-10-20T03:36:19.192458+0000","last_active":"2023-10-20T03:36:20.215015+0000","last_peered":"2023-10-20T03:36:20.215015+0000","last_clean":"2023-10-20T03:36:20.215015+0000","last_became_active":"2023-10-20T03:36:15.163308+0000","last_became_peered":"2023-10-20T03:36:15.163308+0000","last_unstale":"2023-10-20T03:36:20.215015+0000","last_undegraded":"2023-10-20T03:36:20.215015+0000","last_fullsized":"2023-10-20T03:36:20.215015+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003677802,"stat_sum":{"num_bytes":1622016,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":10,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,6,1,14,10,0],"acting":[13,11,6,1,14,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"2"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+undersized","last_fresh":"2023-10-20T03:36:20.224035+0000","last_change":"2023-10-20T03:36:17.183542+0000","last_active":"2023-10-20T03:36:20.224035+0000","last_peered":"2023-10-20T03:36:20.224035+0000","last_clean":"2023-10-20T03:35:04.079338+0000","last_became_active":"2023-10-20T03:36:17.183542+0000","last_became_peered":"2023-10-20T03:36:17.183542+0000","last_unstale":"2023-10-20T03:36:20.224035+0000","last_undegraded":"2023-10-20T03:36:20.224035+0000","last_fullsized":"2023-10-20T03:36:17.180839+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:33.501938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"97'86","reported_seq":502,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-20T03:36:20.214679+0000","last_change":"2023-10-20T03:36:19.241679+0000","last_active":"2023-10-20T03:36:20.214679+0000","last_peered":"2023-10-20T03:36:20.214679+0000","last_clean":"2023-10-20T03:36:20.214679+0000","last_became_active":"2023-10-20T03:36:15.165044+0000","last_became_peered":"2023-10-20T03:36:15.165044+0000","last_unstale":"2023-10-20T03:36:20.214679+0000","last_undegraded":"2023-10-20T03:36:20.214679+0000","last_fullsized":"2023-10-20T03:36:20.214679+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:20.890132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052034807000000002,"stat_sum":{"num_bytes":9535488,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":49,"num_read_kb":37331,"num_write":135,"num_write_kb":10320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":38027264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14,6,2,4,0,15],"acting":[13,14,6,2,4,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"2"},{"start":"39","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":80,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-20T03:36:19.196409+0000","last_change":"2023-10-20T03:35:05.199365+0000","last_active":"2023-10-20T03:36:19.196409+0000","last_peered":"2023-10-20T03:36:19.196409+0000","last_clean":"2023-10-20T03:36:19.196409+0000","last_became_active":"2023-10-20T03:35:05.198615+0000","last_became_peered":"2023-10-20T03:35:05.198615+0000","last_unstale":"2023-10-20T03:36:19.196409+0000","last_undegraded":"2023-10-20T03:36:19.196409+0000","last_fullsized":"2023-10-20T03:36:19.196409+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:23.667783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":103546880,"num_objects":62,"num_object_clones":25,"num_object_copies":434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":6,"num_read":385,"num_read_kb":220851,"num_write":880,"num_write_kb":132752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":91504640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172642304,"data_stored":234037248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":954,"ondisk_log_size":954,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":1119296,"data_compressed":16060,"data_compressed_allocated":69632,"data_compressed_original":1101824,"omap_allocated":2350,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280531,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43040,"kb_used_data":11856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328800,"statfs":{"total":96636764160,"available":96592691200,"internally_reserved":0,"allocated":12140544,"data_stored":15442815,"data_compressed":151590,"data_compressed_allocated":311296,"data_compressed_original":622592,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":11.994,"5min":11.994,"15min":11.994},"last":0.81000000000000005}]},{"osd":1,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":0.749},{"interface":"front","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.411}]},{"osd":2,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.0179999999999998,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0029999999999999}]},{"osd":3,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.403,"5min":2.403,"15min":2.403},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.825,"5min":1.825,"15min":1.825},"last":1.0900000000000001}]},{"osd":4,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.748,"5min":1.748,"15min":1.748},"last":1.298},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":7,"last update":"Fri Oct 20 03:36:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":2.3719999999999999,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":1.264},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.5129999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8600000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7430000000000001}]},{"osd":10,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"max":{"1min":2.4510000000000001,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":1.758},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.6599999999999999},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.7330000000000001,"5min":2.7330000000000001,"15min":2.7330000000000001},"last":1.3759999999999999}]},{"osd":11,"last update":"Fri Oct 20 03:36:17 2023","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.4940000000000002,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.708},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":1.4379999999999999}]},{"osd":12,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":1.6559999999999999}]},{"osd":14,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.5710000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.343},{"interface":"front","average":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.823,"5min":2.823,"15min":2.823},"last":1.206}]},{"osd":15,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":1.679,"15min":1.679},"min":{"1min":1.038,"5min":1.038,"15min":1.038},"max":{"1min":2.403,"5min":2.403,"15min":2.403},"last":1.802},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.7759999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.5660000000000001}]}]},{"osd":15,"up_from":22,"seq":94489280532,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36252,"kb_used_data":9232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335588,"statfs":{"total":96636764160,"available":96599642112,"internally_reserved":0,"allocated":9453568,"data_stored":11717769,"data_compressed":157095,"data_compressed_allocated":339968,"data_compressed_original":1126400,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.913,"5min":1.913,"15min":1.913},"last":1.0329999999999999}]},{"osd":1,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.63}]},{"osd":2,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.996,"5min":0.996,"15min":0.996},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":0.81399999999999995}]},{"osd":3,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.57999999999999996},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.96599999999999997}]},{"osd":4,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.359,"5min":1.359,"15min":1.359},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":1.212}]},{"osd":5,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.825,"5min":1.825,"15min":1.825},"last":0.68700000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]},{"osd":7,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.956,"5min":1.956,"15min":1.956},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.143}]},{"osd":8,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.3399999999999999,"5min":2.3399999999999999,"15min":2.3399999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"last":1.101}]},{"osd":10,"last update":"Fri Oct 20 03:36:13 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.427,"15min":1.427},"min":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"min":{"1min":1.137,"5min":1.137,"15min":1.137},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.2749999999999999}]},{"osd":14,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.8849999999999998,"5min":2.8849999999999998,"15min":2.8849999999999998},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":3.004,"5min":3.004,"15min":3.004},"last":1.246}]}]},{"osd":14,"up_from":22,"seq":94489280532,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46772,"kb_used_data":15588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325068,"statfs":{"total":96636764160,"available":96588869632,"internally_reserved":0,"allocated":15962112,"data_stored":20231039,"data_compressed":151590,"data_compressed_allocated":311296,"data_compressed_original":622592,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,7,8,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":1.333}]},{"osd":1,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":1.0189999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":3,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.3979999999999999}]},{"osd":4,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.4910000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4470000000000001}]},{"osd":7,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.304,"5min":1.304,"15min":1.304},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":1.655}]},{"osd":8,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.7270000000000001}]},{"osd":10,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":1.5309999999999999}]},{"osd":11,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.2999999999999998,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":1.784},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.6589999999999998,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":1.444}]},{"osd":13,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":3.266,"5min":3.266,"15min":3.266},"last":1.5940000000000001}]},{"osd":15,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.5409999999999999,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":1.625},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":3.2090000000000001,"5min":3.2090000000000001,"15min":3.2090000000000001},"last":1.8280000000000001}]}]},{"osd":5,"up_from":21,"seq":90194313236,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33616,"kb_used_data":6592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338224,"statfs":{"total":96636764160,"available":96602341376,"internally_reserved":0,"allocated":6750208,"data_stored":9000991,"data_compressed":149556,"data_compressed_allocated":307200,"data_compressed_original":614400,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,6,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.86199999999999999}]},{"osd":3,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.032},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.798,"5min":1.798,"15min":1.798},"last":1.0660000000000001}]},{"osd":4,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.528,"5min":1.528,"15min":1.528},"last":0.998},{"interface":"front","average":{"1min":1.052,"5min":1.052,"15min":1.052},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.74399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]},{"osd":7,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.425},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":1.2569999999999999}]},{"osd":8,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":1.2430000000000001}]},{"osd":9,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.351,"15min":1.351},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.766,"5min":2.766,"15min":2.766},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":2.484,"5min":2.484,"15min":2.484},"last":1.351}]},{"osd":10,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.7000000000000002,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":1.119},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.212}]},{"osd":11,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"last":1.325}]},{"osd":12,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":2.6120000000000001,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.3859999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5669999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280531,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47144,"kb_used_data":15960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324696,"statfs":{"total":96636764160,"available":96588488704,"internally_reserved":0,"allocated":16343040,"data_stored":19809151,"data_compressed":151590,"data_compressed_allocated":311296,"data_compressed_original":622592,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,5,7,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":0.68400000000000005}]},{"osd":2,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.091},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":0.70499999999999996}]},{"osd":3,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.95999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":5,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.749}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":7,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.0110000000000001,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":0.86499999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.681}]},{"osd":9,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":0.995},{"interface":"front","average":{"1min":1.351,"5min":1.351,"15min":1.351},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.794,"5min":2.794,"15min":2.794},"last":1.2030000000000001}]},{"osd":11,"last update":"Fri Oct 20 03:36:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.879,"5min":1.879,"15min":1.879},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.2130000000000001}]},{"osd":12,"last update":"Fri Oct 20 03:36:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":1.387},{"interface":"front","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.486}]},{"osd":13,"last update":"Fri Oct 20 03:36:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":1.306},{"interface":"front","average":{"1min":1.327,"5min":1.327,"15min":1.327},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":0.88900000000000001}]},{"osd":14,"last update":"Fri Oct 20 03:36:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1.018},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.153}]},{"osd":15,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":1.0640000000000001}]}]},{"osd":1,"up_from":22,"seq":94489280532,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39032,"kb_used_data":12008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332808,"statfs":{"total":96636764160,"available":96596795392,"internally_reserved":0,"allocated":12296192,"data_stored":14754687,"data_compressed":151590,"data_compressed_allocated":311296,"data_compressed_original":622592,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":0.64500000000000002}]},{"osd":2,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.80000000000000004}]},{"osd":3,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":0.59799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":5,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.6440000000000001,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":0.50800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":7,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.351,"5min":1.351,"15min":1.351},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":1.01}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":9,"last update":"Fri Oct 20 03:36:06 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":4.5659999999999998},{"interface":"front","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":6.8940000000000001}]},{"osd":10,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.665,"15min":1.665},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.7000000000000002,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":3.004,"5min":3.004,"15min":3.004},"last":1.206}]},{"osd":11,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":6.367,"5min":6.367,"15min":6.367},"last":1.161},{"interface":"front","average":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":6.1109999999999998,"5min":6.1109999999999998,"15min":6.1109999999999998},"last":0.996}]},{"osd":12,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":2.6760000000000002,"5min":2.6760000000000002,"15min":2.6760000000000002},"last":1.0740000000000001}]},{"osd":13,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.825,"15min":1.825},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":5.8799999999999999,"5min":5.8799999999999999,"15min":5.8799999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":2.0430000000000001,"5min":2.0430000000000001,"15min":2.0430000000000001},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":6.4619999999999997,"5min":6.4619999999999997,"15min":6.4619999999999997},"last":1.0600000000000001}]},{"osd":15,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":3.056,"5min":3.056,"15min":3.056},"last":1.232},{"interface":"front","average":{"1min":1.99,"5min":1.99,"15min":1.99},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":6.7990000000000004,"5min":6.7990000000000004,"15min":6.7990000000000004},"last":1.0449999999999999}]}]},{"osd":2,"up_from":22,"seq":94489280531,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39032,"kb_used_data":12008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332808,"statfs":{"total":96636764160,"available":96596795392,"internally_reserved":0,"allocated":12296192,"data_stored":13894527,"data_compressed":151590,"data_compressed_allocated":311296,"data_compressed_original":622592,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,7,8,10,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":1.2210000000000001}]},{"osd":1,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":0.70999999999999996}]},{"osd":3,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.54800000000000004}]},{"osd":4,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.86099999999999999}]},{"osd":5,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.89200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.226}]},{"osd":7,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.056},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.002}]},{"osd":8,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.333,"15min":1.333},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":3.1040000000000001,"5min":3.1040000000000001,"15min":3.1040000000000001},"last":1.194}]},{"osd":9,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.573,"5min":2.573,"15min":2.573},"last":1.288},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.393}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":15,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"min":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"max":{"1min":2.4129999999999998,"5min":2.4129999999999998,"15min":2.4129999999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.1539999999999999}]}]},{"osd":3,"up_from":22,"seq":94489280531,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46308,"kb_used_data":15128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325532,"statfs":{"total":96636764160,"available":96589344768,"internally_reserved":0,"allocated":15491072,"data_stored":20142697,"data_compressed":149556,"data_compressed_allocated":307200,"data_compressed_original":614400,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7,8,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.669,"5min":2.669,"15min":2.669},"last":1.103},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.91500000000000004}]},{"osd":2,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":1.2130000000000001}]},{"osd":4,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":1.042},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.80500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":7,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.9239999999999999,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":0.94299999999999995}]},{"osd":8,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.1279999999999999}]},{"osd":9,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.622},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.7450000000000001,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.63400000000000001}]},{"osd":10,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.238},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":1.756,"5min":1.756,"15min":1.756},"last":1.3120000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2869999999999999}]},{"osd":14,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":2.5859999999999999,"5min":2.5859999999999999,"15min":2.5859999999999999},"last":1.254},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.306}]},{"osd":15,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.0430000000000001,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":1.425}]}]},{"osd":4,"up_from":22,"seq":94489280532,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49184,"kb_used_data":18000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322656,"statfs":{"total":96636764160,"available":96586399744,"internally_reserved":0,"allocated":18432000,"data_stored":26436479,"data_compressed":151590,"data_compressed_allocated":311296,"data_compressed_original":622592,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":3,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.0029999999999999}]},{"osd":5,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.2889999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":8,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.268},{"interface":"front","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.3919999999999999,"5min":2.3919999999999999,"15min":2.3919999999999999},"last":0.84399999999999997}]},{"osd":9,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.54},{"interface":"front","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":3.2269999999999999,"5min":3.2269999999999999,"15min":3.2269999999999999},"last":1.4039999999999999}]},{"osd":10,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.6599999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":6.7599999999999998,"5min":6.7599999999999998,"15min":6.7599999999999998},"last":1.179},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.407,"5min":2.407,"15min":2.407},"last":1.4339999999999999}]},{"osd":11,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":3.3479999999999999,"5min":3.3479999999999999,"15min":3.3479999999999999},"last":1.173},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":1.365}]},{"osd":12,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":2.589,"5min":2.589,"15min":2.589},"last":1.234},{"interface":"front","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":3.3090000000000002,"5min":3.3090000000000002,"15min":3.3090000000000002},"last":1.135}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4770000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.863}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.323}]}]},{"osd":6,"up_from":89,"seq":382252089347,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40536,"kb_used_data":12172,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94331304,"statfs":{"total":96636764160,"available":96595255296,"internally_reserved":0,"allocated":12464128,"data_stored":15370722,"data_compressed":158077,"data_compressed_allocated":344064,"data_compressed_original":1204224,"omap_allocated":6749,"internal_metadata":29025699},"hb_peers":[0,1,3,5,7,8,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.516}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.478}]}]},{"osd":7,"up_from":22,"seq":94489280530,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36248,"kb_used_data":9224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335592,"statfs":{"total":96636764160,"available":96599646208,"internally_reserved":0,"allocated":9445376,"data_stored":11264895,"data_compressed":151590,"data_compressed_allocated":311296,"data_compressed_original":622592,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,2,3,4,5,6,8,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":2,"last update":"Fri Oct 20 03:36:14 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":1.2170000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.94599999999999995}]},{"osd":3,"last update":"Fri Oct 20 03:36:14 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":1.107}]},{"osd":4,"last update":"Fri Oct 20 03:36:14 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":1.0309999999999999}]},{"osd":5,"last update":"Fri Oct 20 03:36:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.623},{"interface":"front","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.76600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]},{"osd":8,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.3100000000000001}]},{"osd":9,"last update":"Fri Oct 20 03:36:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.171}]},{"osd":10,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":1.25},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":1.4259999999999999}]},{"osd":11,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.401},{"interface":"front","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.4550000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.377}]},{"osd":14,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.359,"15min":1.359},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.677,"5min":1.677,"15min":1.677},"last":1.516},{"interface":"front","average":{"1min":1.496,"5min":1.496,"15min":1.496},"min":{"1min":1.022,"5min":1.022,"15min":1.022},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":1.3500000000000001}]}]},{"osd":8,"up_from":22,"seq":94489280531,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38000,"kb_used_data":10980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333840,"statfs":{"total":96636764160,"available":96597852160,"internally_reserved":0,"allocated":11243520,"data_stored":14207593,"data_compressed":149556,"data_compressed_allocated":307200,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":0.503},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.63500000000000001}]},{"osd":2,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":3.0920000000000001,"5min":3.0920000000000001,"15min":3.0920000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.68200000000000005}]},{"osd":3,"last update":"Fri Oct 20 03:36:05 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":3.2189999999999999,"5min":3.2189999999999999,"15min":3.2189999999999999},"last":0.77200000000000002}]},{"osd":5,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.196,"15min":1.196},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":2.8450000000000002,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.61299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":7,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":4.298,"5min":4.298,"15min":4.298},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.9399999999999999,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.95699999999999996}]},{"osd":9,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.2669999999999999}]},{"osd":10,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":3.02,"5min":3.02,"15min":3.02},"last":1.04}]},{"osd":11,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.476,"15min":1.476},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.218,"5min":2.218,"15min":2.218},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.726,"5min":1.726,"15min":1.726},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":5.141,"5min":5.141,"15min":5.141},"last":0.85499999999999998}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":13,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.25,"5min":2.25,"15min":2.25},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.7429999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":0.90900000000000003}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":15,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":3.2669999999999999,"5min":3.2669999999999999,"15min":3.2669999999999999},"last":1.002},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":4.569,"5min":4.569,"15min":4.569},"last":0.98399999999999999}]}]},{"osd":9,"up_from":22,"seq":94489280529,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34732,"kb_used_data":7712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337108,"statfs":{"total":96636764160,"available":96601198592,"internally_reserved":0,"allocated":7897088,"data_stored":9760977,"data_compressed":131269,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[2,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":2,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":4.2530000000000001,"5min":4.2530000000000001,"15min":4.2530000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.6299999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":0.499}]},{"osd":3,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.9790000000000001,"5min":3.9790000000000001,"15min":3.9790000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":5.0789999999999997,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":0.51700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":8,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.7429999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":3.6659999999999999,"5min":3.6659999999999999,"15min":3.6659999999999999},"last":1.196}]},{"osd":10,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":5.7859999999999996,"5min":5.7859999999999996,"15min":5.7859999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.679,"5min":1.679,"15min":1.679},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":6.9870000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":1.252}]},{"osd":11,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.476,"15min":1.476},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":5.1559999999999997,"5min":5.1559999999999997,"15min":5.1559999999999997},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":5.2999999999999998,"5min":5.2999999999999998,"15min":5.2999999999999998},"last":0.999}]},{"osd":12,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":4.9820000000000002,"5min":4.9820000000000002,"15min":4.9820000000000002},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.327,"5min":1.327,"15min":1.327},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.8199999999999998,"5min":2.8199999999999998,"15min":2.8199999999999998},"last":0.63100000000000001}]},{"osd":13,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":8.6479999999999997,"5min":8.6479999999999997,"15min":8.6479999999999997},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":1.8999999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":8.1170000000000009,"5min":8.1170000000000009,"15min":8.1170000000000009},"last":0.57099999999999995}]},{"osd":14,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":5.5199999999999996,"5min":5.5199999999999996,"15min":5.5199999999999996},"last":0.879},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":3.1320000000000001,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":0.72299999999999998}]},{"osd":15,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.798,"15min":1.798},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":4.7919999999999998,"5min":4.7919999999999998,"15min":4.7919999999999998},"last":1.409},{"interface":"front","average":{"1min":1.738,"5min":1.738,"15min":1.738},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":5.6989999999999998,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":1.0409999999999999}]}]},{"osd":10,"up_from":22,"seq":94489280530,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41680,"kb_used_data":14648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330160,"statfs":{"total":96636764160,"available":96594083840,"internally_reserved":0,"allocated":14999552,"data_stored":17939049,"data_compressed":149556,"data_compressed_allocated":307200,"data_compressed_original":614400,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,8,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.86899999999999999}]},{"osd":1,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.528,"5min":2.528,"15min":2.528},"last":2.1850000000000001},{"interface":"front","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":4.7679999999999998,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":1.4159999999999999}]},{"osd":2,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.621}]},{"osd":3,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.492}]},{"osd":8,"last update":"Fri Oct 20 03:36:13 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":1.762},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"last":1.325}]},{"osd":9,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.464,"15min":1.464},"min":{"1min":1.071,"5min":1.071,"15min":1.071},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.78,"5min":1.78,"15min":1.78},"last":1.641}]},{"osd":11,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":1.857}]},{"osd":12,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":1.115,"5min":1.115,"15min":1.115},"max":{"1min":3.8559999999999999,"5min":3.8559999999999999,"15min":3.8559999999999999},"last":1.958},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":2.0990000000000002}]},{"osd":13,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":2.8260000000000001,"5min":2.8260000000000001,"15min":2.8260000000000001},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"min":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"max":{"1min":3.7069999999999999,"5min":3.7069999999999999,"15min":3.7069999999999999},"last":1.1919999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2390000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.407}]}]},{"osd":11,"up_from":22,"seq":94489280532,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45044,"kb_used_data":13864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326796,"statfs":{"total":96636764160,"available":96590639104,"internally_reserved":0,"allocated":14196736,"data_stored":17270794,"data_compressed":149556,"data_compressed_allocated":307200,"data_compressed_original":614400,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":4.1379999999999999,"5min":4.1379999999999999,"15min":4.1379999999999999},"last":0.91200000000000003}]},{"osd":2,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":4.2290000000000001,"5min":4.2290000000000001,"15min":4.2290000000000001},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Oct 20 03:36:13 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":5.681,"5min":5.681,"15min":5.681},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.782,"5min":1.782,"15min":1.782},"last":0.72399999999999998}]},{"osd":4,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":5.8070000000000004,"5min":5.8070000000000004,"15min":5.8070000000000004},"last":1.234},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.111}]},{"osd":5,"last update":"Fri Oct 20 03:36:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":5.7190000000000003,"5min":5.7190000000000003,"15min":5.7190000000000003},"last":1.139}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":10,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":1.355},{"interface":"front","average":{"1min":1.631,"5min":1.631,"15min":1.631},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":7.2610000000000001,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":1.278}]},{"osd":12,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":9.1959999999999997,"5min":9.1959999999999997,"15min":9.1959999999999997},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.1659999999999999}]},{"osd":13,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":1.02,"5min":1.02,"15min":1.02},"max":{"1min":4.258,"5min":4.258,"15min":4.258},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.643,"5min":1.643,"15min":1.643},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":5.8360000000000003,"5min":5.8360000000000003,"15min":5.8360000000000003},"last":1.3839999999999999}]},{"osd":14,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.544,"15min":1.544},"min":{"1min":1.131,"5min":1.131,"15min":1.131},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.258},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":1.161,"5min":1.161,"15min":1.161},"max":{"1min":4.2830000000000004,"5min":4.2830000000000004,"15min":4.2830000000000004},"last":1.3280000000000001}]},{"osd":15,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"min":{"1min":1.179,"5min":1.179,"15min":1.179},"max":{"1min":5.7809999999999997,"5min":5.7809999999999997,"15min":5.7809999999999997},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.21}]}]},{"osd":12,"up_from":0,"seq":94489280530,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":8521,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":15212544,"data_stored":18960384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11165696,"data_stored":13905920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11165696,"data_stored":13045760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14385152,"data_stored":22196224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":17301504,"data_stored":25587712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5636096,"data_stored":8163328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":11345920,"data_stored":14168064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8314880,"data_stored":10416128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10125312,"data_stored":15220736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":6889472,"data_stored":9019392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":13881344,"data_stored":18952192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":13094912,"data_stored":17473536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":11010048,"data_stored":14594048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":14831616,"data_stored":19382272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":8282112,"data_stored":12951552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T03:36:21.430 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 16 2023-10-20T03:36:21.430 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:21.420+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:21.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-10-20T03:36:21.525 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:21.524+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:21.553 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:21.553 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:21.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-10-20T03:36:21.624 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-10-20T03:36:21.687 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:21.688 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:21.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-10-20T03:36:21.748 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:21.747+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:21.820 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:21.821 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:21.827 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:21.824+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:21.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-10-20T03:36:21.929 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:36:21.927+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:36:22.003 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:22.003 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:22.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-10-20T03:36:22.025 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:36:22.029 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:22.024+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:22.130 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:22.124+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:22.180 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:22.180 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:22.183 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:36:22.183 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-10-20T03:36:22.203 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:36:22.203 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.218 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 3 pgp_num to 16 2023-10-20T03:36:22.222 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:36:22.222 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.229 INFO:tasks.rados.rados.0.smithi170.stdout:365: done (1 left) 2023-10-20T03:36:22.229 INFO:tasks.rados.rados.0.smithi170.stdout:366: snap_create 2023-10-20T03:36:22.229 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 20 v 61 (ObjNum 143 snap 31 seq_num 143) dirty exists 2023-10-20T03:36:22.240 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:22.239+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:22.252 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:36:22.252 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.269 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:36:22.269 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.289 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:36:22.290 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.306 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2023-10-20T03:36:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.326 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:36:22.326 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.331 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:22.324+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:22.334 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:22.334 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:22.342 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:36:22.342 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.343 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.9 2023-10-20T03:36:22.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-10-20T03:36:22.362 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:36:22.362 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.382 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:36:22.383 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.399 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:36:22.399 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.420 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:36:22.421 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.432 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:22.428+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:22.441 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:36:22.441 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.462 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:36:22.460+0000 7f79477fe640 0 -- 172.21.15.33:0/1557479122 >> [v2:172.21.15.170:6819/826332016,v1:172.21.15.170:6822/826332016] conn(0x7f7948220e80 msgr2=0x7f7948223270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-10-20T03:36:22.463 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:36:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:36:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:36:22.467 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:36:22.515 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-10-20T03:36:22.515 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2023-10-20T03:36:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2023-10-20T03:36:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2023-10-20T03:36:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.516 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:36:22.517 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.329666+0000", 2023-10-20T03:36:22.517 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.182585666999998, 2023-10-20T03:36:22.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026825213000000001, 2023-10-20T03:36:22.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.518 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:22.518 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329666+0000", 2023-10-20T03:36:22.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329666+0000", 2023-10-20T03:36:22.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.329667+0000", 2023-10-20T03:36:22.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2023-10-20T03:36:22.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330048+0000", 2023-10-20T03:36:22.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038081299999999997 2023-10-20T03:36:22.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330082+0000", 2023-10-20T03:36:22.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4402e-05 2023-10-20T03:36:22.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330100+0000", 2023-10-20T03:36:22.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7683e-05 2023-10-20T03:36:22.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330268+0000", 2023-10-20T03:36:22.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016784499999999999 2023-10-20T03:36:22.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.330296+0000", 2023-10-20T03:36:22.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8367000000000001e-05 2023-10-20T03:36:22.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356426+0000", 2023-10-20T03:36:22.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026129432000000001 2023-10-20T03:36:22.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.356492+0000", 2023-10-20T03:36:22.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6063999999999996e-05 2023-10-20T03:36:22.529 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.529 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.530 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.530 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.6s1 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:36:22.530 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:25.357167+0000", 2023-10-20T03:36:22.531 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.155084782000003, 2023-10-20T03:36:22.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025761703, 2023-10-20T03:36:22.531 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.531 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:22.531 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357167+0000", 2023-10-20T03:36:22.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357167+0000", 2023-10-20T03:36:22.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357169+0000", 2023-10-20T03:36:22.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.967e-06 2023-10-20T03:36:22.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357532+0000", 2023-10-20T03:36:22.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036283700000000002 2023-10-20T03:36:22.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357564+0000", 2023-10-20T03:36:22.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1798999999999998e-05 2023-10-20T03:36:22.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357615+0000", 2023-10-20T03:36:22.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1004999999999997e-05 2023-10-20T03:36:22.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357832+0000", 2023-10-20T03:36:22.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000216608 2023-10-20T03:36:22.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.357857+0000", 2023-10-20T03:36:22.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5092999999999998e-05 2023-10-20T03:36:22.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.382875+0000", 2023-10-20T03:36:22.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025017978999999999 2023-10-20T03:36:22.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:25.382929+0000", 2023-10-20T03:36:22.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4415e-05 2023-10-20T03:36:22.542 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.542 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.543 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.543 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/31 ECSubWrite(tid=246, reqid=client.4570.0:539, at_version=64'63, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:36:22.543 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.898724+0000", 2023-10-20T03:36:22.544 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.613527859999998, 2023-10-20T03:36:22.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030365014999999999, 2023-10-20T03:36:22.544 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.544 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:22.544 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898724+0000", 2023-10-20T03:36:22.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898724+0000", 2023-10-20T03:36:22.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.898725+0000", 2023-10-20T03:36:22.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.46e-07 2023-10-20T03:36:22.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899207+0000", 2023-10-20T03:36:22.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048263100000000002 2023-10-20T03:36:22.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899243+0000", 2023-10-20T03:36:22.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5139999999999999e-05 2023-10-20T03:36:22.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899266+0000", 2023-10-20T03:36:22.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3142999999999998e-05 2023-10-20T03:36:22.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899450+0000", 2023-10-20T03:36:22.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018391999999999999 2023-10-20T03:36:22.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.899477+0000", 2023-10-20T03:36:22.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7889e-05 2023-10-20T03:36:22.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.928985+0000", 2023-10-20T03:36:22.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029507230999999998 2023-10-20T03:36:22.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.929089+0000", 2023-10-20T03:36:22.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104615 2023-10-20T03:36:22.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.555 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.556 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/31 ECSubWrite(tid=247, reqid=client.4570.0:540, at_version=64'64, trim_to=0'0, roll_forward_to=61'61))", 2023-10-20T03:36:22.556 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:45.913105+0000", 2023-10-20T03:36:22.557 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.599147508999998, 2023-10-20T03:36:22.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028811000999999999, 2023-10-20T03:36:22.557 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.557 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:22.557 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913105+0000", 2023-10-20T03:36:22.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913105+0000", 2023-10-20T03:36:22.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913106+0000", 2023-10-20T03:36:22.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6530000000000001e-06 2023-10-20T03:36:22.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913264+0000", 2023-10-20T03:36:22.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015735400000000001 2023-10-20T03:36:22.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913297+0000", 2023-10-20T03:36:22.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3099000000000003e-05 2023-10-20T03:36:22.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.913312+0000", 2023-10-20T03:36:22.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5713e-05 2023-10-20T03:36:22.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.916852+0000", 2023-10-20T03:36:22.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003540007 2023-10-20T03:36:22.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.916885+0000", 2023-10-20T03:36:22.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2716000000000003e-05 2023-10-20T03:36:22.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.941775+0000", 2023-10-20T03:36:22.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024890223999999999 2023-10-20T03:36:22.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:45.941916+0000", 2023-10-20T03:36:22.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140235 2023-10-20T03:36:22.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.570 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es1 71/31 ECSubWrite(tid=52, reqid=client.4570.0:589, at_version=71'24, trim_to=0'0, roll_forward_to=61'23))", 2023-10-20T03:36:22.570 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:53.004322+0000", 2023-10-20T03:36:22.571 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.507930486999999, 2023-10-20T03:36:22.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044306533000000002, 2023-10-20T03:36:22.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.571 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:22.571 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004322+0000", 2023-10-20T03:36:22.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004322+0000", 2023-10-20T03:36:22.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004324+0000", 2023-10-20T03:36:22.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.187e-06 2023-10-20T03:36:22.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004869+0000", 2023-10-20T03:36:22.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054558099999999995 2023-10-20T03:36:22.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004900+0000", 2023-10-20T03:36:22.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0366999999999999e-05 2023-10-20T03:36:22.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.004937+0000", 2023-10-20T03:36:22.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7509000000000003e-05 2023-10-20T03:36:22.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.005151+0000", 2023-10-20T03:36:22.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021337 2023-10-20T03:36:22.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.005177+0000", 2023-10-20T03:36:22.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6650000000000001e-05 2023-10-20T03:36:22.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.048582+0000", 2023-10-20T03:36:22.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043404953000000003 2023-10-20T03:36:22.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.048628+0000", 2023-10-20T03:36:22.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5915999999999999e-05 2023-10-20T03:36:22.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.582 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.583 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.es1 71/31 ECSubWrite(tid=53, reqid=client.4570.0:590, at_version=71'25, trim_to=0'0, roll_forward_to=61'23))", 2023-10-20T03:36:22.583 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:53.014260+0000", 2023-10-20T03:36:22.583 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.497992013000001, 2023-10-20T03:36:22.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034398564999999999, 2023-10-20T03:36:22.584 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.584 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:22.584 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014260+0000", 2023-10-20T03:36:22.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014260+0000", 2023-10-20T03:36:22.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014262+0000", 2023-10-20T03:36:22.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8470000000000001e-06 2023-10-20T03:36:22.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014803+0000", 2023-10-20T03:36:22.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054058100000000005 2023-10-20T03:36:22.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014844+0000", 2023-10-20T03:36:22.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1383e-05 2023-10-20T03:36:22.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.014867+0000", 2023-10-20T03:36:22.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3116e-05 2023-10-20T03:36:22.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.018959+0000", 2023-10-20T03:36:22.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040918090000000001 2023-10-20T03:36:22.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.018986+0000", 2023-10-20T03:36:22.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7353999999999999e-05 2023-10-20T03:36:22.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.048635+0000", 2023-10-20T03:36:22.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029648364 2023-10-20T03:36:22.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:53.048659+0000", 2023-10-20T03:36:22.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4111e-05 2023-10-20T03:36:22.595 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.595 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.595 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.596 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWrite(3.1s2 77/31 ECSubWrite(tid=303, reqid=client.4570.0:632, at_version=77'73, trim_to=0'0, roll_forward_to=68'72))", 2023-10-20T03:36:22.596 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:35:58.785608+0000", 2023-10-20T03:36:22.597 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.726644193999999, 2023-10-20T03:36:22.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037712515000000002, 2023-10-20T03:36:22.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.597 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:22.597 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.785608+0000", 2023-10-20T03:36:22.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.785608+0000", 2023-10-20T03:36:22.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.785610+0000", 2023-10-20T03:36:22.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.72e-06 2023-10-20T03:36:22.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.788264+0000", 2023-10-20T03:36:22.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026540629999999999 2023-10-20T03:36:22.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.788295+0000", 2023-10-20T03:36:22.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1510000000000002e-05 2023-10-20T03:36:22.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.788311+0000", 2023-10-20T03:36:22.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5739999999999998e-05 2023-10-20T03:36:22.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.788487+0000", 2023-10-20T03:36:22.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176157 2023-10-20T03:36:22.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.788522+0000", 2023-10-20T03:36:22.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4700000000000003e-05 2023-10-20T03:36:22.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.823243+0000", 2023-10-20T03:36:22.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034721152999999998 2023-10-20T03:36:22.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:35:58.823320+0000", 2023-10-20T03:36:22.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7472000000000006e-05 2023-10-20T03:36:22.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.609 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.610 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:781 3.2s0 3:4f9c85d4:::smithi17039300-39:head [append 0~0] snapc 39=[39,38,37,36] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2023-10-20T03:36:22.610 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:13.124299+0000", 2023-10-20T03:36:22.610 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.3879533849999994, 2023-10-20T03:36:22.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025289488999999998, 2023-10-20T03:36:22.610 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.611 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:22.611 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:22.611 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:22.611 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:22.611 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 781 2023-10-20T03:36:22.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.612 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124299+0000", 2023-10-20T03:36:22.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124299+0000", 2023-10-20T03:36:22.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124302+0000", 2023-10-20T03:36:22.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.117e-06 2023-10-20T03:36:22.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124314+0000", 2023-10-20T03:36:22.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1916000000000001e-05 2023-10-20T03:36:22.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124350+0000", 2023-10-20T03:36:22.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5948999999999998e-05 2023-10-20T03:36:22.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124367+0000", 2023-10-20T03:36:22.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7312999999999998e-05 2023-10-20T03:36:22.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124518+0000", 2023-10-20T03:36:22.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015137000000000001 2023-10-20T03:36:22.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2023-10-20T03:36:22.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124535+0000", 2023-10-20T03:36:22.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6929e-05 2023-10-20T03:36:22.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.132271+0000", 2023-10-20T03:36:22.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077359580000000002 2023-10-20T03:36:22.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:22.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.132552+0000", 2023-10-20T03:36:22.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028091200000000002 2023-10-20T03:36:22.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.135165+0000", 2023-10-20T03:36:22.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026132569999999999 2023-10-20T03:36:22.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.143149+0000", 2023-10-20T03:36:22.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079839249999999994 2023-10-20T03:36:22.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:22.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.149335+0000", 2023-10-20T03:36:22.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0061859870000000004 2023-10-20T03:36:22.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.149588+0000", 2023-10-20T03:36:22.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025285599999999998 2023-10-20T03:36:22.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.627 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.628 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:782 3.2s0 3:4f9c85d4:::smithi17039300-39:head [rollback 37] snapc 39=[39,38,37,36] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2023-10-20T03:36:22.628 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:13.124412+0000", 2023-10-20T03:36:22.628 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.3878401270000005, 2023-10-20T03:36:22.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028725628, 2023-10-20T03:36:22.629 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.629 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:22.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:22.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:22.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:22.630 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 782 2023-10-20T03:36:22.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.630 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124412+0000", 2023-10-20T03:36:22.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124412+0000", 2023-10-20T03:36:22.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124413+0000", 2023-10-20T03:36:22.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5600000000000004e-07 2023-10-20T03:36:22.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124420+0000", 2023-10-20T03:36:22.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.323e-06 2023-10-20T03:36:22.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124436+0000", 2023-10-20T03:36:22.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5636000000000001e-05 2023-10-20T03:36:22.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124447+0000", 2023-10-20T03:36:22.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0807e-05 2023-10-20T03:36:22.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124745+0000", 2023-10-20T03:36:22.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029878899999999999 2023-10-20T03:36:22.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:22.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124777+0000", 2023-10-20T03:36:22.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1322000000000003e-05 2023-10-20T03:36:22.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.136698+0000", 2023-10-20T03:36:22.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011920984000000001 2023-10-20T03:36:22.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:22.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.137001+0000", 2023-10-20T03:36:22.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030286499999999998 2023-10-20T03:36:22.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.138685+0000", 2023-10-20T03:36:22.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016848570000000001 2023-10-20T03:36:22.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.144644+0000", 2023-10-20T03:36:22.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059585619999999997 2023-10-20T03:36:22.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:22.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.152961+0000", 2023-10-20T03:36:22.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0083170210000000008 2023-10-20T03:36:22.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.153138+0000", 2023-10-20T03:36:22.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176606 2023-10-20T03:36:22.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.646 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.647 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:783 3.2s0 3:4f9c85d4:::smithi17039300-39:head [append 0~0] snapc 39=[39,38,37,36] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2023-10-20T03:36:22.647 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:13.124487+0000", 2023-10-20T03:36:22.647 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.3877653030000001, 2023-10-20T03:36:22.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030469586, 2023-10-20T03:36:22.647 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.648 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:22.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:22.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:22.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:22.648 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 783 2023-10-20T03:36:22.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.649 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124487+0000", 2023-10-20T03:36:22.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124487+0000", 2023-10-20T03:36:22.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124488+0000", 2023-10-20T03:36:22.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0999999999999998e-07 2023-10-20T03:36:22.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124492+0000", 2023-10-20T03:36:22.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1969999999999998e-06 2023-10-20T03:36:22.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124504+0000", 2023-10-20T03:36:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1956e-05 2023-10-20T03:36:22.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124512+0000", 2023-10-20T03:36:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.106e-06 2023-10-20T03:36:22.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124846+0000", 2023-10-20T03:36:22.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000334445 2023-10-20T03:36:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.124859+0000", 2023-10-20T03:36:22.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3179999999999999e-05 2023-10-20T03:36:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.140289+0000", 2023-10-20T03:36:22.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015429506000000001 2023-10-20T03:36:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:22.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.140617+0000", 2023-10-20T03:36:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000327762 2023-10-20T03:36:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.142313+0000", 2023-10-20T03:36:22.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001696576 2023-10-20T03:36:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.147591+0000", 2023-10-20T03:36:22.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052780689999999998 2023-10-20T03:36:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.154689+0000", 2023-10-20T03:36:22.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070977499999999999 2023-10-20T03:36:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:13.154956+0000", 2023-10-20T03:36:22.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026732900000000001 2023-10-20T03:36:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:824 3.2s0 3:474cd252:::smithi17039300-42:head [append 0~802816] snapc 3b=[39,38] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2023-10-20T03:36:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:18.219540+0000", 2023-10-20T03:36:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2927122620000002, 2023-10-20T03:36:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039651001999999998, 2023-10-20T03:36:22.670 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:22.671 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 824 2023-10-20T03:36:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.219540+0000", 2023-10-20T03:36:22.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.219540+0000", 2023-10-20T03:36:22.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.219543+0000", 2023-10-20T03:36:22.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9229999999999998e-06 2023-10-20T03:36:22.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.220970+0000", 2023-10-20T03:36:22.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014269370000000001 2023-10-20T03:36:22.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.221003+0000", 2023-10-20T03:36:22.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3593000000000002e-05 2023-10-20T03:36:22.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.221019+0000", 2023-10-20T03:36:22.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5438999999999999e-05 2023-10-20T03:36:22.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.221146+0000", 2023-10-20T03:36:22.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012767099999999999 2023-10-20T03:36:22.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:22.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.221362+0000", 2023-10-20T03:36:22.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000216045 2023-10-20T03:36:22.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.223582+0000", 2023-10-20T03:36:22.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022195330000000001 2023-10-20T03:36:22.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.257857+0000", 2023-10-20T03:36:22.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034274517999999997 2023-10-20T03:36:22.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:22.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.258984+0000", 2023-10-20T03:36:22.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001127315 2023-10-20T03:36:22.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.259191+0000", 2023-10-20T03:36:22.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000207028 2023-10-20T03:36:22.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.684 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.685 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.685 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:825 3.2s0 3:474cd252:::smithi17039300-42:head [append 0~425984] snapc 3b=[39,38] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2023-10-20T03:36:22.685 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:18.221888+0000", 2023-10-20T03:36:22.685 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.290363782, 2023-10-20T03:36:22.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039312518999999997, 2023-10-20T03:36:22.686 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.686 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:22.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:22.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:22.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:22.687 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 825 2023-10-20T03:36:22.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.687 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.221888+0000", 2023-10-20T03:36:22.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.221888+0000", 2023-10-20T03:36:22.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.221890+0000", 2023-10-20T03:36:22.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.426e-06 2023-10-20T03:36:22.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.222454+0000", 2023-10-20T03:36:22.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056383400000000004 2023-10-20T03:36:22.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.222478+0000", 2023-10-20T03:36:22.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4076000000000001e-05 2023-10-20T03:36:22.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.222489+0000", 2023-10-20T03:36:22.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1677e-05 2023-10-20T03:36:22.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.243407+0000", 2023-10-20T03:36:22.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020917188 2023-10-20T03:36:22.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:22.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.243663+0000", 2023-10-20T03:36:22.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025658400000000002 2023-10-20T03:36:22.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.245391+0000", 2023-10-20T03:36:22.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017278739999999999 2023-10-20T03:36:22.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.259573+0000", 2023-10-20T03:36:22.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014182468 2023-10-20T03:36:22.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:22.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.261063+0000", 2023-10-20T03:36:22.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014890789999999999 2023-10-20T03:36:22.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.261201+0000", 2023-10-20T03:36:22.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013831299999999999 2023-10-20T03:36:22.698 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.698 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.699 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:826 3.2s0 3:474cd252:::smithi17039300-42:head [append 0~720896] snapc 3b=[39,38] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2023-10-20T03:36:22.699 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:18.225939+0000", 2023-10-20T03:36:22.699 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.286312841, 2023-10-20T03:36:22.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062966066000000001, 2023-10-20T03:36:22.700 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.700 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:22.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:22.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:22.701 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:22.701 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 826 2023-10-20T03:36:22.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.701 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.225939+0000", 2023-10-20T03:36:22.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.225939+0000", 2023-10-20T03:36:22.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.225941+0000", 2023-10-20T03:36:22.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1900000000000002e-06 2023-10-20T03:36:22.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.226959+0000", 2023-10-20T03:36:22.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010170940000000001 2023-10-20T03:36:22.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.226990+0000", 2023-10-20T03:36:22.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1375999999999999e-05 2023-10-20T03:36:22.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.227007+0000", 2023-10-20T03:36:22.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7005999999999999e-05 2023-10-20T03:36:22.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.262074+0000", 2023-10-20T03:36:22.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035067478999999999 2023-10-20T03:36:22.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:22.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.262432+0000", 2023-10-20T03:36:22.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035792499999999999 2023-10-20T03:36:22.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.265005+0000", 2023-10-20T03:36:22.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002572572 2023-10-20T03:36:22.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.287430+0000", 2023-10-20T03:36:22.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022425457999999999 2023-10-20T03:36:22.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:22.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.288745+0000", 2023-10-20T03:36:22.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00131456 2023-10-20T03:36:22.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.288905+0000", 2023-10-20T03:36:22.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016040600000000001 2023-10-20T03:36:22.712 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.712 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.712 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.713 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:827 3.2s0 3:474cd252:::smithi17039300-42:head [append 0~409600] snapc 3b=[39,38] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2023-10-20T03:36:22.713 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:18.228207+0000", 2023-10-20T03:36:22.713 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2840453060000003, 2023-10-20T03:36:22.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063666674000000006, 2023-10-20T03:36:22.714 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.714 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:22.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:22.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:22.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:22.715 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 827 2023-10-20T03:36:22.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.715 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228207+0000", 2023-10-20T03:36:22.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228207+0000", 2023-10-20T03:36:22.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228208+0000", 2023-10-20T03:36:22.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.237e-06 2023-10-20T03:36:22.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228809+0000", 2023-10-20T03:36:22.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060045300000000003 2023-10-20T03:36:22.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228833+0000", 2023-10-20T03:36:22.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4406e-05 2023-10-20T03:36:22.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228846+0000", 2023-10-20T03:36:22.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3006e-05 2023-10-20T03:36:22.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.278120+0000", 2023-10-20T03:36:22.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049274387000000003 2023-10-20T03:36:22.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:22.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.278331+0000", 2023-10-20T03:36:22.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021033900000000001 2023-10-20T03:36:22.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.279731+0000", 2023-10-20T03:36:22.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001400185 2023-10-20T03:36:22.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.289286+0000", 2023-10-20T03:36:22.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0095549799999999994 2023-10-20T03:36:22.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:22.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.291692+0000", 2023-10-20T03:36:22.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024062480000000002 2023-10-20T03:36:22.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.291874+0000", 2023-10-20T03:36:22.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000181433 2023-10-20T03:36:22.726 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.726 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.727 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:828 3.2s0 3:474cd252:::smithi17039300-42:head [setxattr _header (57) in=7b] snapc 3b=[39,38] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2023-10-20T03:36:22.727 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:18.228870+0000", 2023-10-20T03:36:22.728 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2833820840000003, 2023-10-20T03:36:22.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.066167711000000004, 2023-10-20T03:36:22.728 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.728 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:22.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:22.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:22.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:22.729 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 828 2023-10-20T03:36:22.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.729 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228870+0000", 2023-10-20T03:36:22.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228870+0000", 2023-10-20T03:36:22.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228871+0000", 2023-10-20T03:36:22.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7000000000000003e-07 2023-10-20T03:36:22.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228876+0000", 2023-10-20T03:36:22.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3360000000000002e-06 2023-10-20T03:36:22.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228891+0000", 2023-10-20T03:36:22.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4715999999999999e-05 2023-10-20T03:36:22.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228899+0000", 2023-10-20T03:36:22.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.14e-06 2023-10-20T03:36:22.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.289493+0000", 2023-10-20T03:36:22.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060593586999999997 2023-10-20T03:36:22.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.289684+0000", 2023-10-20T03:36:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191106 2023-10-20T03:36:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.290492+0000", 2023-10-20T03:36:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080838100000000005 2023-10-20T03:36:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.293970+0000", 2023-10-20T03:36:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034772309999999999 2023-10-20T03:36:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.294820+0000", 2023-10-20T03:36:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085003599999999998 2023-10-20T03:36:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.295038+0000", 2023-10-20T03:36:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021820800000000001 2023-10-20T03:36:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:829 3.2s0 3:474cd252:::smithi17039300-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2023-10-20T03:36:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:18.228912+0000", 2023-10-20T03:36:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2833401919999998, 2023-10-20T03:36:22.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071597870999999993, 2023-10-20T03:36:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2023-10-20T03:36:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:22.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 829 2023-10-20T03:36:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.745 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228912+0000", 2023-10-20T03:36:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228912+0000", 2023-10-20T03:36:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228913+0000", 2023-10-20T03:36:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4e-07 2023-10-20T03:36:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228916+0000", 2023-10-20T03:36:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6430000000000001e-06 2023-10-20T03:36:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228930+0000", 2023-10-20T03:36:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3833e-05 2023-10-20T03:36:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228938+0000", 2023-10-20T03:36:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9100000000000005e-06 2023-10-20T03:36:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.291108+0000", 2023-10-20T03:36:22.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062169783999999999 2023-10-20T03:36:22.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.291297+0000", 2023-10-20T03:36:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018888999999999999 2023-10-20T03:36:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.295206+0000", 2023-10-20T03:36:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039087540000000004 2023-10-20T03:36:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:22.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.295534+0000", 2023-10-20T03:36:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032848600000000001 2023-10-20T03:36:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.300510+0000", 2023-10-20T03:36:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0049757309999999997 2023-10-20T03:36:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:830 3.2s0 3:4f9c85d4:::smithi17039300-39:head [append 0~0] snapc 3b=[39,38] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2023-10-20T03:36:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:18.228950+0000", 2023-10-20T03:36:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2833019830000003, 2023-10-20T03:36:22.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.068817680000000006, 2023-10-20T03:36:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:22.758 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 830 2023-10-20T03:36:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228950+0000", 2023-10-20T03:36:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228950+0000", 2023-10-20T03:36:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228951+0000", 2023-10-20T03:36:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1699999999999997e-07 2023-10-20T03:36:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228954+0000", 2023-10-20T03:36:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2260000000000001e-06 2023-10-20T03:36:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228967+0000", 2023-10-20T03:36:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2627e-05 2023-10-20T03:36:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228978+0000", 2023-10-20T03:36:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0890000000000001e-05 2023-10-20T03:36:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.291993+0000", 2023-10-20T03:36:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063015110999999999 2023-10-20T03:36:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:22.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.292181+0000", 2023-10-20T03:36:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000188009 2023-10-20T03:36:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.293113+0000", 2023-10-20T03:36:22.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093272199999999998 2023-10-20T03:36:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.296815+0000", 2023-10-20T03:36:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037019900000000001 2023-10-20T03:36:22.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.297499+0000", 2023-10-20T03:36:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068349399999999999 2023-10-20T03:36:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.297768+0000", 2023-10-20T03:36:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026889399999999999 2023-10-20T03:36:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.770 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:831 3.2s0 3:4f9c85d4:::smithi17039300-39:head [rollback 38] snapc 3b=[39,38] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2023-10-20T03:36:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:18.228989+0000", 2023-10-20T03:36:22.771 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2832630910000002, 2023-10-20T03:36:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.078916468000000004, 2023-10-20T03:36:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:22.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 831 2023-10-20T03:36:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228989+0000", 2023-10-20T03:36:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228989+0000", 2023-10-20T03:36:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228990+0000", 2023-10-20T03:36:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.37e-07 2023-10-20T03:36:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.228995+0000", 2023-10-20T03:36:22.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.853e-06 2023-10-20T03:36:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.229007+0000", 2023-10-20T03:36:22.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2557e-05 2023-10-20T03:36:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.229014+0000", 2023-10-20T03:36:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2629999999999997e-06 2023-10-20T03:36:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.301458+0000", 2023-10-20T03:36:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.072443478000000006 2023-10-20T03:36:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.301638+0000", 2023-10-20T03:36:22.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000180156 2023-10-20T03:36:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.302668+0000", 2023-10-20T03:36:22.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010294309999999999 2023-10-20T03:36:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.306796+0000", 2023-10-20T03:36:22.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0041283179999999997 2023-10-20T03:36:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.307625+0000", 2023-10-20T03:36:22.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00082889499999999996 2023-10-20T03:36:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.307906+0000", 2023-10-20T03:36:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028078000000000001 2023-10-20T03:36:22.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4570.0:832 3.2s0 3:4f9c85d4:::smithi17039300-39:head [append 0~0] snapc 3b=[39,38] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2023-10-20T03:36:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:18.229025+0000", 2023-10-20T03:36:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2832270210000001, 2023-10-20T03:36:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080306174999999994, 2023-10-20T03:36:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.786 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2023-10-20T03:36:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4570", 2023-10-20T03:36:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 832 2023-10-20T03:36:22.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.229025+0000", 2023-10-20T03:36:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.229025+0000", 2023-10-20T03:36:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.229026+0000", 2023-10-20T03:36:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6599999999999996e-07 2023-10-20T03:36:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.229029+0000", 2023-10-20T03:36:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2200000000000001e-06 2023-10-20T03:36:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.229041+0000", 2023-10-20T03:36:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2323e-05 2023-10-20T03:36:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.229048+0000", 2023-10-20T03:36:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1099999999999997e-06 2023-10-20T03:36:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.303578+0000", 2023-10-20T03:36:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.074529766999999997 2023-10-20T03:36:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2023-10-20T03:36:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.303944+0000", 2023-10-20T03:36:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000366058 2023-10-20T03:36:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_started", 2023-10-20T03:36:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.305533+0000", 2023-10-20T03:36:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001588667 2023-10-20T03:36:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_committed", 2023-10-20T03:36:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.308387+0000", 2023-10-20T03:36:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028538830000000002 2023-10-20T03:36:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2023-10-20T03:36:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.309071+0000", 2023-10-20T03:36:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068466399999999998 2023-10-20T03:36:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.309331+0000", 2023-10-20T03:36:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025981700000000002 2023-10-20T03:36:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.799 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 96/31 ECSubWriteReply(tid=104, last_complete=96'82, committed=1, applied=1))", 2023-10-20T03:36:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2023-10-20T03:36:18.262693+0000", 2023-10-20T03:36:22.800 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2495591529999999, 2023-10-20T03:36:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024869155, 2023-10-20T03:36:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2023-10-20T03:36:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2023-10-20T03:36:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2023-10-20T03:36:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2023-10-20T03:36:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.262693+0000", 2023-10-20T03:36:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2023-10-20T03:36:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.262693+0000", 2023-10-20T03:36:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2023-10-20T03:36:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2023-10-20T03:36:22.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.262694+0000", 2023-10-20T03:36:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.06e-06 2023-10-20T03:36:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2023-10-20T03:36:22.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.262701+0000", 2023-10-20T03:36:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2030000000000003e-06 2023-10-20T03:36:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2023-10-20T03:36:22.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.262724+0000", 2023-10-20T03:36:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2546000000000001e-05 2023-10-20T03:36:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2023-10-20T03:36:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.262733+0000", 2023-10-20T03:36:22.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6770000000000007e-06 2023-10-20T03:36:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2023-10-20T03:36:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.287526+0000", 2023-10-20T03:36:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024792242999999999 2023-10-20T03:36:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-10-20T03:36:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-10-20T03:36:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2023-10-20T03:36:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2023-10-20T03:36:18.287562+0000", 2023-10-20T03:36:22.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6426e-05 2023-10-20T03:36:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-10-20T03:36:22.809 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-10-20T03:36:22.809 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-10-20T03:36:22.811 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-10-20T03:36:22.812 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:22.808+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:22.911 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:22.908+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:22.942 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:22.942 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:36:22.942 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2023-10-20T03:36:22.942 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:22.956 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-10-20T03:36:23.012 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:23.008+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:23.078 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:23.079 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:36:23.079 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:36:23.079 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:36:23.079 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:23.092 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-10-20T03:36:23.113 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:23.108+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:23.214 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:23.214 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2023-10-20T03:36:23.214 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2023-10-20T03:36:23.214 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2023-10-20T03:36:23.214 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.215 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:323 3.0s0 3:0e4cfe35:::smithi17039300-49:head [append 0~589824] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:23.215 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.357685+0000", 2023-10-20T03:36:23.215 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 56.852517173999999, 2023-10-20T03:36:23.215 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039449378, 2023-10-20T03:36:23.215 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.216 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:23.216 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.216 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.216 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.216 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 323 2023-10-20T03:36:23.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.217 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.217 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.217 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.357685+0000", 2023-10-20T03:36:23.218 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.218 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.218 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.218 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.357685+0000", 2023-10-20T03:36:23.218 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.219 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.219 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.219 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.219 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.357688+0000", 2023-10-20T03:36:23.219 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7159999999999999e-06 2023-10-20T03:36:23.220 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.220 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.220 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.220 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358334+0000", 2023-10-20T03:36:23.220 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00064636299999999995 2023-10-20T03:36:23.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.221 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.221 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.221 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358368+0000", 2023-10-20T03:36:23.221 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3182000000000002e-05 2023-10-20T03:36:23.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.222 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.222 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.222 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358382+0000", 2023-10-20T03:36:23.222 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4797000000000001e-05 2023-10-20T03:36:23.222 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.223 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.223 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.223 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358559+0000", 2023-10-20T03:36:23.223 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000177057 2023-10-20T03:36:23.223 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.224 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.224 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.358736+0000", 2023-10-20T03:36:23.224 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000176861 2023-10-20T03:36:23.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.225 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.225 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361012+0000", 2023-10-20T03:36:23.225 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002275792 2023-10-20T03:36:23.225 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.225 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.226 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.226 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.396529+0000", 2023-10-20T03:36:23.226 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.035517254999999998 2023-10-20T03:36:23.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.226 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.226 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.396672+0000", 2023-10-20T03:36:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014286600000000001 2023-10-20T03:36:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.227 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.397135+0000", 2023-10-20T03:36:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00046248900000000001 2023-10-20T03:36:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.228 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:324 3.0s0 3:0e4cfe35:::smithi17039300-49:head [append 0~573440] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.365103+0000", 2023-10-20T03:36:23.229 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 56.845099390000001, 2023-10-20T03:36:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.045538596000000001, 2023-10-20T03:36:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.230 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 324 2023-10-20T03:36:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.231 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365103+0000", 2023-10-20T03:36:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.232 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365103+0000", 2023-10-20T03:36:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.233 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365105+0000", 2023-10-20T03:36:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3e-06 2023-10-20T03:36:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365744+0000", 2023-10-20T03:36:23.234 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000638213 2023-10-20T03:36:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365777+0000", 2023-10-20T03:36:23.235 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3522000000000002e-05 2023-10-20T03:36:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.365792+0000", 2023-10-20T03:36:23.236 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5227e-05 2023-10-20T03:36:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.379267+0000", 2023-10-20T03:36:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013474832000000001 2023-10-20T03:36:23.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.379462+0000", 2023-10-20T03:36:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019448100000000001 2023-10-20T03:36:23.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.381551+0000", 2023-10-20T03:36:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002089034 2023-10-20T03:36:23.239 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.404299+0000", 2023-10-20T03:36:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.022748244000000001 2023-10-20T03:36:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.240 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.410190+0000", 2023-10-20T03:36:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0058907630000000002 2023-10-20T03:36:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.241 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.410642+0000", 2023-10-20T03:36:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00045197999999999999 2023-10-20T03:36:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.242 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:325 3.0s0 3:0e4cfe35:::smithi17039300-49:head [append 0~196608] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.367541+0000", 2023-10-20T03:36:23.243 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 56.842661237000002, 2023-10-20T03:36:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.044806372999999997, 2023-10-20T03:36:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.244 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 325 2023-10-20T03:36:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.245 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367541+0000", 2023-10-20T03:36:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.246 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367541+0000", 2023-10-20T03:36:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.247 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367544+0000", 2023-10-20T03:36:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6129999999999998e-06 2023-10-20T03:36:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.248 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367692+0000", 2023-10-20T03:36:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014814900000000001 2023-10-20T03:36:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367725+0000", 2023-10-20T03:36:23.249 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2752000000000001e-05 2023-10-20T03:36:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367759+0000", 2023-10-20T03:36:23.250 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4654999999999999e-05 2023-10-20T03:36:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.397445+0000", 2023-10-20T03:36:23.251 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029685547 2023-10-20T03:36:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.397646+0000", 2023-10-20T03:36:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020082300000000001 2023-10-20T03:36:23.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:36:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001263 2023-10-20T03:36:23.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.405032+0000", 2023-10-20T03:36:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0061230240000000003 2023-10-20T03:36:23.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.412065+0000", 2023-10-20T03:36:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.007033437 2023-10-20T03:36:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.412348+0000", 2023-10-20T03:36:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00028237299999999999 2023-10-20T03:36:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.256 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:326 3.0s0 3:0e4cfe35:::smithi17039300-49:head [setxattr _header (56) in=7b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:23.257 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.367791+0000", 2023-10-20T03:36:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 56.842411878999997, 2023-10-20T03:36:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.045708658999999999, 2023-10-20T03:36:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.258 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 326 2023-10-20T03:36:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.259 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367791+0000", 2023-10-20T03:36:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367791+0000", 2023-10-20T03:36:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.261 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367792+0000", 2023-10-20T03:36:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1370000000000001e-06 2023-10-20T03:36:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.262 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367798+0000", 2023-10-20T03:36:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1229999999999997e-06 2023-10-20T03:36:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367813+0000", 2023-10-20T03:36:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5130000000000001e-05 2023-10-20T03:36:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.264 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367823+0000", 2023-10-20T03:36:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0103e-05 2023-10-20T03:36:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.265 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402753+0000", 2023-10-20T03:36:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.034930226000000002 2023-10-20T03:36:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.266 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.402936+0000", 2023-10-20T03:36:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000182542 2023-10-20T03:36:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.403933+0000", 2023-10-20T03:36:23.267 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00099678799999999997 2023-10-20T03:36:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.405527+0000", 2023-10-20T03:36:23.268 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001594332 2023-10-20T03:36:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.413173+0000", 2023-10-20T03:36:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0076458400000000001 2023-10-20T03:36:23.269 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.270 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.270 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.270 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.413499+0000", 2023-10-20T03:36:23.270 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00032643800000000002 2023-10-20T03:36:23.270 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.271 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.271 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.271 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.271 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:327 3.0s0 3:0e4cfe35:::smithi17039300-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:23.271 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.367835+0000", 2023-10-20T03:36:23.272 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 56.842367123000002, 2023-10-20T03:36:23.272 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.053408074999999999, 2023-10-20T03:36:23.272 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.272 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:23.272 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.273 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.273 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.273 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 327 2023-10-20T03:36:23.273 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.273 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.274 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.274 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367835+0000", 2023-10-20T03:36:23.274 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.275 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.275 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367835+0000", 2023-10-20T03:36:23.275 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.275 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.275 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.276 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.276 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367836+0000", 2023-10-20T03:36:23.276 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3699999999999999e-07 2023-10-20T03:36:23.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.276 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.277 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367840+0000", 2023-10-20T03:36:23.277 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6200000000000001e-06 2023-10-20T03:36:23.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.277 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.278 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367851+0000", 2023-10-20T03:36:23.278 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0764999999999999e-05 2023-10-20T03:36:23.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.279 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.279 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.367858+0000", 2023-10-20T03:36:23.279 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7700000000000001e-06 2023-10-20T03:36:23.279 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.279 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.280 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.280 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.404384+0000", 2023-10-20T03:36:23.280 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.036525558999999999 2023-10-20T03:36:23.280 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.280 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.280 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:23.281 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.404538+0000", 2023-10-20T03:36:23.281 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000154 2023-10-20T03:36:23.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.281 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.282 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.414622+0000", 2023-10-20T03:36:23.282 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.01008392 2023-10-20T03:36:23.282 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.282 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.282 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.283 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.414717+0000", 2023-10-20T03:36:23.283 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5476999999999994e-05 2023-10-20T03:36:23.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.283 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.283 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.284 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.421243+0000", 2023-10-20T03:36:23.284 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0065260270000000002 2023-10-20T03:36:23.284 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.284 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.284 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.284 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.285 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.285 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:390 3.0s0 3:0e4cfe35:::smithi17039300-49:head [copy-from ver 15] snapc f=[f,d,b,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:36:23.285 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:31.382452+0000", 2023-10-20T03:36:23.285 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 51.827750795999997, 2023-10-20T03:36:23.285 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.059495292999999998, 2023-10-20T03:36:23.286 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.286 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:23.286 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.286 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.286 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.287 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 390 2023-10-20T03:36:23.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.287 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.287 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.288 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382452+0000", 2023-10-20T03:36:23.288 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.288 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.288 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.288 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.289 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382452+0000", 2023-10-20T03:36:23.289 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.289 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.289 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382455+0000", 2023-10-20T03:36:23.290 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4070000000000002e-06 2023-10-20T03:36:23.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.290 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.290 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382474+0000", 2023-10-20T03:36:23.291 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8422000000000001e-05 2023-10-20T03:36:23.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.291 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.291 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382516+0000", 2023-10-20T03:36:23.291 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2376000000000002e-05 2023-10-20T03:36:23.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382534+0000", 2023-10-20T03:36:23.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8335999999999999e-05 2023-10-20T03:36:23.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382662+0000", 2023-10-20T03:36:23.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000128083 2023-10-20T03:36:23.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.294 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op must wait for map", 2023-10-20T03:36:23.294 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382676+0000", 2023-10-20T03:36:23.294 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3679999999999999e-05 2023-10-20T03:36:23.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.295 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.295 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.387857+0000", 2023-10-20T03:36:23.295 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0051807490000000001 2023-10-20T03:36:23.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.296 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.296 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.388142+0000", 2023-10-20T03:36:23.296 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000284729 2023-10-20T03:36:23.296 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.296 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.296 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.297 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.413968+0000", 2023-10-20T03:36:23.297 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025826947999999999 2023-10-20T03:36:23.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.297 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.297 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.440223+0000", 2023-10-20T03:36:23.298 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026254116000000001 2023-10-20T03:36:23.298 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.298 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.298 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:23.298 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.441418+0000", 2023-10-20T03:36:23.298 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001195113 2023-10-20T03:36:23.299 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.299 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.299 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.299 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.441947+0000", 2023-10-20T03:36:23.299 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00052933400000000001 2023-10-20T03:36:23.300 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.300 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.300 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.300 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.300 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.300 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:391 3.0s0 3:0e4cfe35:::smithi17039300-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2023-10-20T03:36:23.301 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:31.382572+0000", 2023-10-20T03:36:23.301 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 51.827630563, 2023-10-20T03:36:23.301 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.056308376, 2023-10-20T03:36:23.301 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.301 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:23.301 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.302 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.302 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.302 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 391 2023-10-20T03:36:23.302 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.302 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.302 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.303 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.303 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382572+0000", 2023-10-20T03:36:23.303 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.303 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.303 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.304 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.304 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382572+0000", 2023-10-20T03:36:23.304 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.304 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.304 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.305 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382573+0000", 2023-10-20T03:36:23.305 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4130000000000001e-06 2023-10-20T03:36:23.305 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.305 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.305 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.306 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382580+0000", 2023-10-20T03:36:23.306 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0670000000000003e-06 2023-10-20T03:36:23.306 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.306 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.306 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.307 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382603+0000", 2023-10-20T03:36:23.307 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2099999999999998e-05 2023-10-20T03:36:23.307 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.307 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.307 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.307 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382612+0000", 2023-10-20T03:36:23.308 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3519999999999996e-06 2023-10-20T03:36:23.308 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.308 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.308 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.308 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382806+0000", 2023-10-20T03:36:23.309 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019382199999999999 2023-10-20T03:36:23.309 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.309 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.309 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:23.309 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.382827+0000", 2023-10-20T03:36:23.309 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1552e-05 2023-10-20T03:36:23.310 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.310 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.310 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.310 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.388779+0000", 2023-10-20T03:36:23.310 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0059519960000000002 2023-10-20T03:36:23.311 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.311 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.311 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for blocked object", 2023-10-20T03:36:23.311 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.388904+0000", 2023-10-20T03:36:23.311 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000124893 2023-10-20T03:36:23.312 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.312 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.312 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.312 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.438358+0000", 2023-10-20T03:36:23.312 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.049453814999999998 2023-10-20T03:36:23.312 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.313 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.313 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.313 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.438533+0000", 2023-10-20T03:36:23.313 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017465199999999999 2023-10-20T03:36:23.313 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.314 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.314 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.314 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.438880+0000", 2023-10-20T03:36:23.314 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000347714 2023-10-20T03:36:23.314 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.315 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.315 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.315 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.315 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.315 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:408 3.fs1 3:f90c0d61:::smithi17039300-26:head [append 0~753664] snapc f=[d,3,2] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:36:23.316 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.443873+0000", 2023-10-20T03:36:23.316 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 47.766329704999997, 2023-10-20T03:36:23.316 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.045012336, 2023-10-20T03:36:23.316 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.316 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:23.317 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.317 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.317 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.317 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 408 2023-10-20T03:36:23.317 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.318 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.318 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.318 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.318 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.443873+0000", 2023-10-20T03:36:23.318 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.319 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.319 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.319 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.319 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.443873+0000", 2023-10-20T03:36:23.319 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.319 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.320 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.320 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.320 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.443876+0000", 2023-10-20T03:36:23.320 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8200000000000001e-06 2023-10-20T03:36:23.320 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.321 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.321 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.321 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.444508+0000", 2023-10-20T03:36:23.321 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00063252700000000002 2023-10-20T03:36:23.321 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.321 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.322 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.322 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.444542+0000", 2023-10-20T03:36:23.322 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4292999999999998e-05 2023-10-20T03:36:23.322 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.322 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.323 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.323 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.444564+0000", 2023-10-20T03:36:23.323 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1066000000000001e-05 2023-10-20T03:36:23.323 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.323 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.323 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.324 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.446120+0000", 2023-10-20T03:36:23.324 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0015568089999999999 2023-10-20T03:36:23.324 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.324 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.324 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.325 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.446540+0000", 2023-10-20T03:36:23.325 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00041965900000000002 2023-10-20T03:36:23.325 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.325 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.325 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.325 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.449079+0000", 2023-10-20T03:36:23.326 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0025392610000000001 2023-10-20T03:36:23.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.326 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.326 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.326 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.469859+0000", 2023-10-20T03:36:23.327 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.020779766000000002 2023-10-20T03:36:23.327 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.327 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.327 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:23.327 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.488709+0000", 2023-10-20T03:36:23.327 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.018849798000000001 2023-10-20T03:36:23.328 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.328 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.328 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.328 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.488885+0000", 2023-10-20T03:36:23.328 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017633699999999999 2023-10-20T03:36:23.328 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.329 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.329 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.329 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.329 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.329 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:409 3.fs1 3:f90c0d61:::smithi17039300-26:head [append 0~720896] snapc f=[d,3,2] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:36:23.329 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.456955+0000", 2023-10-20T03:36:23.330 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 47.753247647999999, 2023-10-20T03:36:23.330 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039289445999999999, 2023-10-20T03:36:23.330 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.330 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:23.330 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.331 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.331 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.331 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 409 2023-10-20T03:36:23.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.331 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.331 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.332 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.332 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.456955+0000", 2023-10-20T03:36:23.332 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.332 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.332 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.333 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.456955+0000", 2023-10-20T03:36:23.333 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.333 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.333 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.333 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.334 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.456958+0000", 2023-10-20T03:36:23.334 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.253e-06 2023-10-20T03:36:23.334 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.334 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.334 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.335 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.457987+0000", 2023-10-20T03:36:23.335 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0010284510000000001 2023-10-20T03:36:23.335 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.335 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.335 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.335 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458025+0000", 2023-10-20T03:36:23.336 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8566000000000003e-05 2023-10-20T03:36:23.336 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.336 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.336 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.336 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458043+0000", 2023-10-20T03:36:23.337 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.783e-05 2023-10-20T03:36:23.337 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.337 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.337 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.337 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.470966+0000", 2023-10-20T03:36:23.338 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012923244 2023-10-20T03:36:23.338 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.338 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.338 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.338 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.471149+0000", 2023-10-20T03:36:23.339 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000182638 2023-10-20T03:36:23.339 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.339 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.339 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.339 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.472881+0000", 2023-10-20T03:36:23.340 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017317610000000001 2023-10-20T03:36:23.340 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.340 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.340 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.340 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.488407+0000", 2023-10-20T03:36:23.340 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015526521 2023-10-20T03:36:23.341 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.341 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.341 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:23.341 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.496153+0000", 2023-10-20T03:36:23.341 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0077454489999999997 2023-10-20T03:36:23.342 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.342 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.342 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.342 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.496244+0000", 2023-10-20T03:36:23.342 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1732999999999997e-05 2023-10-20T03:36:23.343 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.343 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.343 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.343 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.343 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.344 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:410 3.fs1 3:f90c0d61:::smithi17039300-26:head [append 0~49152] snapc f=[d,3,2] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:36:23.344 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.458073+0000", 2023-10-20T03:36:23.344 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 47.752129463000003, 2023-10-20T03:36:23.344 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039280043000000001, 2023-10-20T03:36:23.344 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.344 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:23.345 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.345 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.345 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.345 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 410 2023-10-20T03:36:23.345 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.346 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.346 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.346 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.346 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458073+0000", 2023-10-20T03:36:23.346 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.346 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.347 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.347 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.347 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458073+0000", 2023-10-20T03:36:23.347 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.347 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.347 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.348 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.348 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458074+0000", 2023-10-20T03:36:23.348 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2100000000000001e-06 2023-10-20T03:36:23.348 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.348 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.348 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.349 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458151+0000", 2023-10-20T03:36:23.349 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6542e-05 2023-10-20T03:36:23.349 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.349 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.349 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.349 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458168+0000", 2023-10-20T03:36:23.350 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7552999999999999e-05 2023-10-20T03:36:23.350 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.350 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.350 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.350 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458180+0000", 2023-10-20T03:36:23.351 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1826e-05 2023-10-20T03:36:23.351 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.351 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.351 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.351 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.485589+0000", 2023-10-20T03:36:23.351 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027409103000000001 2023-10-20T03:36:23.352 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.352 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.352 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.352 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.485726+0000", 2023-10-20T03:36:23.352 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000136482 2023-10-20T03:36:23.352 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.353 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.353 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.353 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.486676+0000", 2023-10-20T03:36:23.353 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000949766 2023-10-20T03:36:23.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.354 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.354 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.354 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.491210+0000", 2023-10-20T03:36:23.354 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0045347230000000001 2023-10-20T03:36:23.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.355 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.355 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:23.355 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.497284+0000", 2023-10-20T03:36:23.355 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0060733059999999997 2023-10-20T03:36:23.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.356 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.356 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.356 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.497353+0000", 2023-10-20T03:36:23.356 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9531999999999998e-05 2023-10-20T03:36:23.356 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.357 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.357 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.357 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.357 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.357 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:411 3.fs1 3:f90c0d61:::smithi17039300-26:head [setxattr _header (57) in=7b] snapc f=[d,3,2] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:36:23.357 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.458196+0000", 2023-10-20T03:36:23.358 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 47.752006561999998, 2023-10-20T03:36:23.358 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039654613999999998, 2023-10-20T03:36:23.358 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.358 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:23.358 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.359 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.359 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.359 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 411 2023-10-20T03:36:23.359 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.359 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.360 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.360 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.360 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458196+0000", 2023-10-20T03:36:23.360 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.360 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.361 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.361 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.361 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458196+0000", 2023-10-20T03:36:23.361 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.361 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.361 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.362 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.362 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458197+0000", 2023-10-20T03:36:23.362 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1299999999999998e-07 2023-10-20T03:36:23.362 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.362 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.363 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.363 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458202+0000", 2023-10-20T03:36:23.363 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2170000000000002e-06 2023-10-20T03:36:23.363 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.363 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.363 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.364 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458212+0000", 2023-10-20T03:36:23.364 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0000000000000001e-05 2023-10-20T03:36:23.364 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.364 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.364 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.365 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458220+0000", 2023-10-20T03:36:23.365 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4560000000000003e-06 2023-10-20T03:36:23.365 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.365 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.365 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.366 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.489002+0000", 2023-10-20T03:36:23.366 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030782868000000001 2023-10-20T03:36:23.366 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.366 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.366 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.366 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.489245+0000", 2023-10-20T03:36:23.367 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024254100000000001 2023-10-20T03:36:23.367 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.367 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.367 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.367 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.490266+0000", 2023-10-20T03:36:23.367 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001020721 2023-10-20T03:36:23.368 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.368 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.368 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.368 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.491971+0000", 2023-10-20T03:36:23.368 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017054940000000001 2023-10-20T03:36:23.368 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.369 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.369 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:23.369 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.497768+0000", 2023-10-20T03:36:23.369 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0057964269999999998 2023-10-20T03:36:23.369 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.370 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.370 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.370 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.497851+0000", 2023-10-20T03:36:23.370 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2977000000000002e-05 2023-10-20T03:36:23.370 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.371 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.371 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.371 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.371 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.371 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:412 3.fs1 3:f90c0d61:::smithi17039300-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2023-10-20T03:36:23.372 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.458230+0000", 2023-10-20T03:36:23.372 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 47.751972696999999, 2023-10-20T03:36:23.372 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.042333239000000002, 2023-10-20T03:36:23.372 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.372 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:23.372 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.373 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.373 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.373 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 412 2023-10-20T03:36:23.373 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.373 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.374 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.374 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.374 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458230+0000", 2023-10-20T03:36:23.374 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.374 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.375 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.375 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.375 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458230+0000", 2023-10-20T03:36:23.375 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.375 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.376 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.376 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.376 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458231+0000", 2023-10-20T03:36:23.376 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8700000000000005e-07 2023-10-20T03:36:23.376 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.377 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.377 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.377 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458234+0000", 2023-10-20T03:36:23.377 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5499999999999999e-06 2023-10-20T03:36:23.377 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.377 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.378 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.378 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458245+0000", 2023-10-20T03:36:23.378 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0716999999999999e-05 2023-10-20T03:36:23.378 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.378 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.378 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.379 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.458252+0000", 2023-10-20T03:36:23.379 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.486e-06 2023-10-20T03:36:23.379 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.379 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.379 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.379 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.491656+0000", 2023-10-20T03:36:23.380 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.033403571 2023-10-20T03:36:23.380 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.380 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.380 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:23.380 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.491757+0000", 2023-10-20T03:36:23.381 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101383 2023-10-20T03:36:23.381 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.381 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.381 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.381 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.498257+0000", 2023-10-20T03:36:23.381 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0064998809999999999 2023-10-20T03:36:23.382 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.382 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.382 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.382 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.498357+0000", 2023-10-20T03:36:23.382 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9491e-05 2023-10-20T03:36:23.383 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.383 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.383 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.383 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.500563+0000", 2023-10-20T03:36:23.383 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0022063730000000002 2023-10-20T03:36:23.384 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.384 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.384 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.384 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.384 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.385 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:421 3.0s0 3:0e4cfe35:::smithi17039300-49:head [setxattr _header (57) in=7b] snapc f=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-10-20T03:36:23.385 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.469730+0000", 2023-10-20T03:36:23.385 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 46.740472918999998, 2023-10-20T03:36:23.385 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.033931731999999999, 2023-10-20T03:36:23.385 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.386 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:23.386 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.386 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.386 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.386 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 421 2023-10-20T03:36:23.387 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.387 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.387 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.387 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.387 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469730+0000", 2023-10-20T03:36:23.387 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.388 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.388 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.388 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.388 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469730+0000", 2023-10-20T03:36:23.388 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.389 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.389 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.389 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.389 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469731+0000", 2023-10-20T03:36:23.389 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5799999999999999e-06 2023-10-20T03:36:23.389 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.390 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.390 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.390 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469741+0000", 2023-10-20T03:36:23.390 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6700000000000006e-06 2023-10-20T03:36:23.390 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.391 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.391 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.391 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469765+0000", 2023-10-20T03:36:23.391 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4216e-05 2023-10-20T03:36:23.391 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.391 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.392 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.392 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469780+0000", 2023-10-20T03:36:23.392 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.482e-05 2023-10-20T03:36:23.392 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.392 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.393 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.393 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.496944+0000", 2023-10-20T03:36:23.393 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027164562999999999 2023-10-20T03:36:23.393 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.393 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.394 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.394 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.497153+0000", 2023-10-20T03:36:23.394 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020828900000000001 2023-10-20T03:36:23.394 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.394 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.394 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.395 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.498227+0000", 2023-10-20T03:36:23.395 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0010742670000000001 2023-10-20T03:36:23.395 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.395 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.395 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.396 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.501137+0000", 2023-10-20T03:36:23.396 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0029095969999999999 2023-10-20T03:36:23.396 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.396 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.396 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:23.397 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.503446+0000", 2023-10-20T03:36:23.397 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0023098580000000001 2023-10-20T03:36:23.397 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.397 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.397 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.397 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.503661+0000", 2023-10-20T03:36:23.398 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00021487200000000001 2023-10-20T03:36:23.398 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.398 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.398 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.398 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.399 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.399 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:422 3.0s0 3:0e4cfe35:::smithi17039300-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2023-10-20T03:36:23.399 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.469801+0000", 2023-10-20T03:36:23.399 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 46.740401880999997, 2023-10-20T03:36:23.399 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.036242191, 2023-10-20T03:36:23.399 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.400 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:23.400 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.400 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.400 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.400 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 422 2023-10-20T03:36:23.400 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.401 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.401 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.401 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.401 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469801+0000", 2023-10-20T03:36:23.401 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.401 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.402 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.402 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.402 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469801+0000", 2023-10-20T03:36:23.402 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.402 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.403 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.403 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.403 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469802+0000", 2023-10-20T03:36:23.403 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.19e-06 2023-10-20T03:36:23.403 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.403 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.404 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.404 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469809+0000", 2023-10-20T03:36:23.404 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4630000000000004e-06 2023-10-20T03:36:23.404 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.404 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.404 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.405 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469832+0000", 2023-10-20T03:36:23.405 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2883000000000001e-05 2023-10-20T03:36:23.405 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.405 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.405 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.406 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.469846+0000", 2023-10-20T03:36:23.406 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4113e-05 2023-10-20T03:36:23.406 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.406 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.406 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.406 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.500041+0000", 2023-10-20T03:36:23.407 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030195161000000002 2023-10-20T03:36:23.407 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.407 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.407 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:23.407 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.500396+0000", 2023-10-20T03:36:23.407 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00035421399999999999 2023-10-20T03:36:23.408 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.408 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.408 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.408 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.504208+0000", 2023-10-20T03:36:23.408 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0038123390000000001 2023-10-20T03:36:23.409 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.409 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.409 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.409 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.504283+0000", 2023-10-20T03:36:23.409 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4981999999999995e-05 2023-10-20T03:36:23.409 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.410 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.410 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.410 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.506043+0000", 2023-10-20T03:36:23.410 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001759846 2023-10-20T03:36:23.410 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.411 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.411 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.411 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.411 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.411 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.1s5 60/31 ECSubWrite(tid=190, reqid=client.4570.0:452, at_version=60'54, trim_to=0'0, roll_forward_to=58'48))", 2023-10-20T03:36:23.412 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.861666+0000", 2023-10-20T03:36:23.412 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 41.348536928000001, 2023-10-20T03:36:23.412 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.037670670000000003, 2023-10-20T03:36:23.412 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.412 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:23.413 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.413 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.413 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.413 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.861666+0000", 2023-10-20T03:36:23.413 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.413 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.414 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.414 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.414 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.861666+0000", 2023-10-20T03:36:23.414 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.415 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.415 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.415 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.415 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.861692+0000", 2023-10-20T03:36:23.415 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6239000000000001e-05 2023-10-20T03:36:23.415 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.416 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.416 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.416 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.862699+0000", 2023-10-20T03:36:23.416 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0010070540000000001 2023-10-20T03:36:23.416 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.417 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.417 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.417 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.862728+0000", 2023-10-20T03:36:23.417 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8626e-05 2023-10-20T03:36:23.417 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.417 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.418 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.418 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.862748+0000", 2023-10-20T03:36:23.418 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0199000000000001e-05 2023-10-20T03:36:23.418 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.418 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.419 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.419 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.862946+0000", 2023-10-20T03:36:23.419 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019865900000000001 2023-10-20T03:36:23.419 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.419 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.419 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.420 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.862974+0000", 2023-10-20T03:36:23.420 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7616000000000001e-05 2023-10-20T03:36:23.420 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.420 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.420 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.420 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.899243+0000", 2023-10-20T03:36:23.421 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.03626923 2023-10-20T03:36:23.421 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.421 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.421 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.421 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.899336+0000", 2023-10-20T03:36:23.422 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3047000000000005e-05 2023-10-20T03:36:23.422 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.422 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.422 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.422 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.423 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.423 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:552 3.0s0 3:083aab2b:::smithi17039300-2:head [copy-from ver 11] snapc 23=[23,21] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2023-10-20T03:36:23.423 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:50.925989+0000", 2023-10-20T03:36:23.423 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 32.284213286000004, 2023-10-20T03:36:23.423 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.061771054999999998, 2023-10-20T03:36:23.424 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.424 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:23.424 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.424 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.424 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.425 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 552 2023-10-20T03:36:23.425 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.425 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.425 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.425 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.425 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.925989+0000", 2023-10-20T03:36:23.426 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.426 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.426 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.426 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.426 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.925989+0000", 2023-10-20T03:36:23.426 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.427 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.427 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.427 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.925992+0000", 2023-10-20T03:36:23.427 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0699999999999998e-06 2023-10-20T03:36:23.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.428 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.428 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.428 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926010+0000", 2023-10-20T03:36:23.428 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8099999999999999e-05 2023-10-20T03:36:23.428 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.428 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.429 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.429 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926045+0000", 2023-10-20T03:36:23.429 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5089000000000002e-05 2023-10-20T03:36:23.429 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.429 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.429 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.430 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926062+0000", 2023-10-20T03:36:23.430 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6897000000000001e-05 2023-10-20T03:36:23.430 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.430 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.430 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.430 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926214+0000", 2023-10-20T03:36:23.431 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015152699999999999 2023-10-20T03:36:23.431 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.431 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.431 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op must wait for map", 2023-10-20T03:36:23.431 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926233+0000", 2023-10-20T03:36:23.431 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8949000000000001e-05 2023-10-20T03:36:23.432 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.432 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.432 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.432 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.931605+0000", 2023-10-20T03:36:23.432 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0053720529999999999 2023-10-20T03:36:23.433 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.433 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.433 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.433 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.931794+0000", 2023-10-20T03:36:23.433 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018947300000000001 2023-10-20T03:36:23.433 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.434 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.434 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.434 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.953907+0000", 2023-10-20T03:36:23.434 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.022112526 2023-10-20T03:36:23.434 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.435 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.435 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.435 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.984803+0000", 2023-10-20T03:36:23.435 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030896078 2023-10-20T03:36:23.435 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.435 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.436 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:23.436 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.987358+0000", 2023-10-20T03:36:23.436 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0025545469999999999 2023-10-20T03:36:23.436 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.436 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.437 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.437 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.987760+0000", 2023-10-20T03:36:23.437 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000402746 2023-10-20T03:36:23.437 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.437 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.437 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.438 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.438 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.438 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:553 3.0s0 3:083aab2b:::smithi17039300-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2023-10-20T03:36:23.438 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:50.926100+0000", 2023-10-20T03:36:23.438 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 32.284102058000002, 2023-10-20T03:36:23.439 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.058318110999999999, 2023-10-20T03:36:23.439 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.439 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:23.439 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.439 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.440 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.440 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 553 2023-10-20T03:36:23.440 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.440 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.440 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.440 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.441 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926100+0000", 2023-10-20T03:36:23.441 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.441 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.441 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.441 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926100+0000", 2023-10-20T03:36:23.442 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.442 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.442 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.442 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.442 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926102+0000", 2023-10-20T03:36:23.442 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3769999999999999e-06 2023-10-20T03:36:23.443 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.443 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.443 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.443 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926109+0000", 2023-10-20T03:36:23.443 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8159999999999996e-06 2023-10-20T03:36:23.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.444 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.444 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.444 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926131+0000", 2023-10-20T03:36:23.444 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2370000000000001e-05 2023-10-20T03:36:23.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.445 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.445 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.445 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926147+0000", 2023-10-20T03:36:23.445 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6025999999999999e-05 2023-10-20T03:36:23.445 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.446 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.446 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.446 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926345+0000", 2023-10-20T03:36:23.446 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019841300000000001 2023-10-20T03:36:23.446 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.447 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.447 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:23.447 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.926358+0000", 2023-10-20T03:36:23.447 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2262999999999999e-05 2023-10-20T03:36:23.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.447 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.448 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.448 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.932162+0000", 2023-10-20T03:36:23.448 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0058040820000000003 2023-10-20T03:36:23.448 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.448 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.448 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for blocked object", 2023-10-20T03:36:23.449 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.932353+0000", 2023-10-20T03:36:23.449 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019135300000000001 2023-10-20T03:36:23.449 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.449 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.449 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.450 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.983370+0000", 2023-10-20T03:36:23.450 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.051016367999999999 2023-10-20T03:36:23.450 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.450 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.450 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.450 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.983685+0000", 2023-10-20T03:36:23.451 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00031522700000000001 2023-10-20T03:36:23.451 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.451 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.451 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.451 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.984419+0000", 2023-10-20T03:36:23.451 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00073381599999999998 2023-10-20T03:36:23.452 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.452 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.452 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.452 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.452 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.453 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.es5 71/31 ECSubWrite(tid=52, reqid=client.4570.0:589, at_version=71'24, trim_to=0'0, roll_forward_to=61'23))", 2023-10-20T03:36:23.453 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.004276+0000", 2023-10-20T03:36:23.453 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 30.205926553000001, 2023-10-20T03:36:23.453 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.033989473999999999, 2023-10-20T03:36:23.453 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.454 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:23.454 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.454 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.454 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.454 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004276+0000", 2023-10-20T03:36:23.455 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.455 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.455 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.455 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.455 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004276+0000", 2023-10-20T03:36:23.455 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.456 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.456 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.456 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.456 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004276+0000", 2023-10-20T03:36:23.456 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4999999999999998e-07 2023-10-20T03:36:23.457 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.457 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.457 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.457 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004557+0000", 2023-10-20T03:36:23.457 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00028076400000000002 2023-10-20T03:36:23.458 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.458 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.458 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.458 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004588+0000", 2023-10-20T03:36:23.458 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0417e-05 2023-10-20T03:36:23.459 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.459 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.459 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.459 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004621+0000", 2023-10-20T03:36:23.459 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3416e-05 2023-10-20T03:36:23.459 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.460 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.460 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.460 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.005126+0000", 2023-10-20T03:36:23.460 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00050556099999999997 2023-10-20T03:36:23.460 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.460 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.461 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.461 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.005192+0000", 2023-10-20T03:36:23.461 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5429e-05 2023-10-20T03:36:23.461 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.461 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.462 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.462 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.038177+0000", 2023-10-20T03:36:23.462 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.032984861999999997 2023-10-20T03:36:23.462 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.462 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.463 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.463 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.038265+0000", 2023-10-20T03:36:23.463 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.8675000000000001e-05 2023-10-20T03:36:23.463 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.463 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.464 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.464 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.464 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.464 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:628 3.0s0 3:0ac18d01:::smithi17039300-46:head [copy-from ver 66] snapc 28=[28,26] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2023-10-20T03:36:23.464 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:57.010408+0000", 2023-10-20T03:36:23.465 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 26.199794376, 2023-10-20T03:36:23.465 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.069670394999999996, 2023-10-20T03:36:23.465 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.465 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:23.465 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.466 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.466 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.466 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 628 2023-10-20T03:36:23.466 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.466 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.467 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.467 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.467 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010408+0000", 2023-10-20T03:36:23.467 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.467 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.468 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.468 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.468 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010408+0000", 2023-10-20T03:36:23.468 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.468 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.468 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.469 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.469 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010411+0000", 2023-10-20T03:36:23.469 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.666e-06 2023-10-20T03:36:23.469 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.469 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.470 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.470 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010423+0000", 2023-10-20T03:36:23.470 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2466999999999999e-05 2023-10-20T03:36:23.470 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.470 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.471 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.471 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010458+0000", 2023-10-20T03:36:23.471 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4224000000000003e-05 2023-10-20T03:36:23.471 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.471 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.472 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.472 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010474+0000", 2023-10-20T03:36:23.472 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6325999999999999e-05 2023-10-20T03:36:23.472 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.472 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.473 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.473 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010589+0000", 2023-10-20T03:36:23.473 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011505700000000001 2023-10-20T03:36:23.473 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.473 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.473 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.474 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010780+0000", 2023-10-20T03:36:23.474 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000191246 2023-10-20T03:36:23.474 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.474 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.474 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.475 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.036134+0000", 2023-10-20T03:36:23.475 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025353844 2023-10-20T03:36:23.475 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.475 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.475 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.476 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.076512+0000", 2023-10-20T03:36:23.476 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.040378485999999998 2023-10-20T03:36:23.476 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.476 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.476 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:23.476 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.079728+0000", 2023-10-20T03:36:23.477 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0032154549999999999 2023-10-20T03:36:23.477 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.477 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.477 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.477 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.080079+0000", 2023-10-20T03:36:23.478 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00035062400000000001 2023-10-20T03:36:23.478 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.478 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.478 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.478 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.479 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.479 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:629 3.0s0 3:0ac18d01:::smithi17039300-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2023-10-20T03:36:23.479 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:57.010513+0000", 2023-10-20T03:36:23.479 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 26.199690015000002, 2023-10-20T03:36:23.479 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.065291919000000004, 2023-10-20T03:36:23.479 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.480 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:23.480 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:23.480 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:23.480 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:23.480 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 629 2023-10-20T03:36:23.480 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.481 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.481 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.481 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.481 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010513+0000", 2023-10-20T03:36:23.481 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.481 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.482 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.482 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.482 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010513+0000", 2023-10-20T03:36:23.482 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.482 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.483 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.483 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.483 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010514+0000", 2023-10-20T03:36:23.483 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0899999999999999e-06 2023-10-20T03:36:23.483 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.484 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.484 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.484 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010519+0000", 2023-10-20T03:36:23.484 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2630000000000003e-06 2023-10-20T03:36:23.484 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.485 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.485 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.485 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010532+0000", 2023-10-20T03:36:23.485 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2989e-05 2023-10-20T03:36:23.485 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.486 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.486 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.486 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.010541+0000", 2023-10-20T03:36:23.486 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9129999999999992e-06 2023-10-20T03:36:23.486 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.487 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.487 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.487 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.011068+0000", 2023-10-20T03:36:23.487 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00052740100000000004 2023-10-20T03:36:23.487 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.487 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.488 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for blocked object", 2023-10-20T03:36:23.488 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.011176+0000", 2023-10-20T03:36:23.488 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010788700000000001 2023-10-20T03:36:23.488 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.488 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.489 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.489 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.075253+0000", 2023-10-20T03:36:23.489 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.064077032000000006 2023-10-20T03:36:23.489 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.489 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.489 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:23.490 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.075450+0000", 2023-10-20T03:36:23.490 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019677799999999999 2023-10-20T03:36:23.490 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.490 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.490 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.491 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.075804+0000", 2023-10-20T03:36:23.491 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00035456600000000001 2023-10-20T03:36:23.491 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.491 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.491 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.492 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.492 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.492 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:23.492 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-10-20T03:36:23.493 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-10-20T03:36:23.494 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:23.212+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:23.495 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-10-20T03:36:23.498 INFO:tasks.rados.rados.0.smithi170.stdout:363: done (1 left) 2023-10-20T03:36:23.499 INFO:tasks.rados.rados.0.smithi170.stdout:366: done (0 left) 2023-10-20T03:36:23.499 INFO:tasks.rados.rados.0.smithi170.stdout:367: copy_from oid 18 from oid 2 current snap is 33 2023-10-20T03:36:23.499 INFO:tasks.rados.rados.0.smithi170.stdout:368: delete oid 9 current snap is 33 2023-10-20T03:36:23.499 INFO:tasks.rados.rados.0.smithi170.stdout:368: done (1 left) 2023-10-20T03:36:23.499 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 35 current snap is 33 2023-10-20T03:36:23.500 INFO:tasks.rados.rados.0.smithi170.stdout:369: seq_num 149 ranges {0=98304} 2023-10-20T03:36:23.500 INFO:tasks.rados.rados.0.smithi170.stdout:369: writing smithi17039300-35 from 0 to 98304 tid 1 2023-10-20T03:36:23.500 INFO:tasks.rados.rados.0.smithi170.stdout:370: read oid 38 snap -1 2023-10-20T03:36:23.500 INFO:tasks.rados.rados.0.smithi170.stdout:370: expect (ObjNum 96 snap 13 seq_num 96) 2023-10-20T03:36:23.500 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 25 current snap is 33 2023-10-20T03:36:23.501 INFO:tasks.rados.rados.0.smithi170.stdout:371: seq_num 150 ranges {2490368=114688} 2023-10-20T03:36:23.501 INFO:tasks.rados.rados.0.smithi170.stdout:371: writing smithi17039300-25 from 2490368 to 2605056 tid 1 2023-10-20T03:36:23.501 INFO:tasks.rados.rados.0.smithi170.stdout:372: copy_from oid 49 from oid 6 current snap is 33 2023-10-20T03:36:23.501 INFO:tasks.rados.rados.0.smithi170.stdout:373: setattr oid 19 current snap is 33 2023-10-20T03:36:23.501 INFO:tasks.rados.rados.0.smithi170.stdout:370: expect (ObjNum 96 snap 13 seq_num 96) 2023-10-20T03:36:23.501 INFO:tasks.rados.rados.0.smithi170.stdout:369: finishing write tid 1 to smithi17039300-35 2023-10-20T03:36:23.502 INFO:tasks.rados.rados.0.smithi170.stdout:372: finishing copy_from to smithi17039300-49 2023-10-20T03:36:23.502 INFO:tasks.rados.rados.0.smithi170.stdout:372: got expected ENOENT (src dne) 2023-10-20T03:36:23.502 INFO:tasks.rados.rados.0.smithi170.stdout:372: finishing copy_from racing read to smithi17039300-49 2023-10-20T03:36:23.502 INFO:tasks.rados.rados.0.smithi170.stdout:367: finishing copy_from racing read to smithi17039300-18 2023-10-20T03:36:23.502 INFO:tasks.rados.rados.0.smithi170.stdout:369: finishing write tid 2 to smithi17039300-35 2023-10-20T03:36:23.503 INFO:tasks.rados.rados.0.smithi170.stdout:371: finishing write tid 1 to smithi17039300-25 2023-10-20T03:36:23.503 INFO:tasks.rados.rados.0.smithi170.stdout:369: finishing write tid 3 to smithi17039300-35 2023-10-20T03:36:23.503 INFO:tasks.rados.rados.0.smithi170.stdout:369: oid 35 updating version 0 to 67 2023-10-20T03:36:23.503 INFO:tasks.rados.rados.0.smithi170.stdout:369: oid 35 updating version 67 to 68 2023-10-20T03:36:23.503 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 68 (ObjNum 149 snap 33 seq_num 149) dirty exists 2023-10-20T03:36:23.504 INFO:tasks.rados.rados.0.smithi170.stdout:369: left oid 35 (ObjNum 149 snap 33 seq_num 149) 2023-10-20T03:36:23.504 INFO:tasks.rados.rados.0.smithi170.stdout:371: finishing write tid 2 to smithi17039300-25 2023-10-20T03:36:23.504 INFO:tasks.rados.rados.0.smithi170.stdout:371: finishing write tid 3 to smithi17039300-25 2023-10-20T03:36:23.504 INFO:tasks.rados.rados.0.smithi170.stdout:371: oid 25 updating version 0 to 56 2023-10-20T03:36:23.504 INFO:tasks.rados.rados.0.smithi170.stdout:371: oid 25 version 56 is already newer than 55 2023-10-20T03:36:23.505 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 25 v 56 (ObjNum 150 snap 33 seq_num 150) dirty exists 2023-10-20T03:36:23.505 INFO:tasks.rados.rados.0.smithi170.stdout:371: left oid 25 (ObjNum 150 snap 33 seq_num 150) 2023-10-20T03:36:23.505 INFO:tasks.rados.rados.0.smithi170.stdout:367: finishing copy_from to smithi17039300-18 2023-10-20T03:36:23.505 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 18 v 11 (ObjNum 96 snap 13 seq_num 96) dirty exists 2023-10-20T03:36:23.505 INFO:tasks.rados.rados.0.smithi170.stdout:367: done (5 left) 2023-10-20T03:36:23.506 INFO:tasks.rados.rados.0.smithi170.stdout:369: done (4 left) 2023-10-20T03:36:23.506 INFO:tasks.rados.rados.0.smithi170.stdout:370: done (3 left) 2023-10-20T03:36:23.506 INFO:tasks.rados.rados.0.smithi170.stdout:371: done (2 left) 2023-10-20T03:36:23.506 INFO:tasks.rados.rados.0.smithi170.stdout:372: done (1 left) 2023-10-20T03:36:23.506 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 21 current snap is 33 2023-10-20T03:36:23.506 INFO:tasks.rados.rados.0.smithi170.stdout:374: seq_num 151 ranges {4374528=131072} 2023-10-20T03:36:23.507 INFO:tasks.rados.rados.0.smithi170.stdout:374: writing smithi17039300-21 from 4374528 to 4505600 tid 1 2023-10-20T03:36:23.507 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 22 current snap is 33 2023-10-20T03:36:23.507 INFO:tasks.rados.rados.0.smithi170.stdout:375: seq_num 152 ranges {360448=147456} 2023-10-20T03:36:23.507 INFO:tasks.rados.rados.0.smithi170.stdout:375: writing smithi17039300-22 from 360448 to 507904 tid 1 2023-10-20T03:36:23.507 INFO:tasks.rados.rados.0.smithi170.stdout:376: snap_create 2023-10-20T03:36:23.508 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 63 (ObjNum 120 snap 25 seq_num 120) dirty exists 2023-10-20T03:36:23.508 INFO:tasks.rados.rados.0.smithi170.stdout:374: finishing write tid 1 to smithi17039300-21 2023-10-20T03:36:23.508 INFO:tasks.rados.rados.0.smithi170.stdout:374: finishing write tid 2 to smithi17039300-21 2023-10-20T03:36:23.508 INFO:tasks.rados.rados.0.smithi170.stdout:375: finishing write tid 1 to smithi17039300-22 2023-10-20T03:36:23.508 INFO:tasks.rados.rados.0.smithi170.stdout:374: finishing write tid 3 to smithi17039300-21 2023-10-20T03:36:23.508 INFO:tasks.rados.rados.0.smithi170.stdout:374: oid 21 updating version 0 to 80 2023-10-20T03:36:23.509 INFO:tasks.rados.rados.0.smithi170.stdout:374: oid 21 version 80 is already newer than 79 2023-10-20T03:36:23.509 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 21 v 80 (ObjNum 151 snap 33 seq_num 151) dirty exists 2023-10-20T03:36:23.509 INFO:tasks.rados.rados.0.smithi170.stdout:374: left oid 21 (ObjNum 151 snap 33 seq_num 151) 2023-10-20T03:36:23.509 INFO:tasks.rados.rados.0.smithi170.stdout:375: finishing write tid 2 to smithi17039300-22 2023-10-20T03:36:23.509 INFO:tasks.rados.rados.0.smithi170.stdout:375: finishing write tid 3 to smithi17039300-22 2023-10-20T03:36:23.510 INFO:tasks.rados.rados.0.smithi170.stdout:375: oid 22 updating version 0 to 66 2023-10-20T03:36:23.510 INFO:tasks.rados.rados.0.smithi170.stdout:375: oid 22 version 66 is already newer than 65 2023-10-20T03:36:23.510 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 66 (ObjNum 152 snap 33 seq_num 152) dirty exists 2023-10-20T03:36:23.510 INFO:tasks.rados.rados.0.smithi170.stdout:375: left oid 22 (ObjNum 152 snap 33 seq_num 152) 2023-10-20T03:36:23.596 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:23.592+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:23.640 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:23.640 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:36:23.641 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2023-10-20T03:36:23.641 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:23.654 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-10-20T03:36:23.704 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:23.700+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:23.775 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:23.775 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:36:23.775 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:36:23.775 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:36:23.775 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:23.788 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-10-20T03:36:23.797 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:36:23.795+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:36:23.898 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:23.892+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:23.959 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:23.959 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2023-10-20T03:36:23.959 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2023-10-20T03:36:23.959 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2023-10-20T03:36:23.959 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.960 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.2s6 36/31 ECSubWrite(tid=21, reqid=client.4570.0:198, at_version=36'13, trim_to=0'0, roll_forward_to=35'11))", 2023-10-20T03:36:23.960 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:18.266756+0000", 2023-10-20T03:36:23.960 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 65.689062927999998, 2023-10-20T03:36:23.960 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025327286000000001, 2023-10-20T03:36:23.961 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.961 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:23.961 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.961 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.961 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.962 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.266756+0000", 2023-10-20T03:36:23.962 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.962 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.962 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.962 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.962 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.266756+0000", 2023-10-20T03:36:23.963 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.963 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.963 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.963 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.963 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.266756+0000", 2023-10-20T03:36:23.964 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4299999999999999e-07 2023-10-20T03:36:23.964 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.964 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.964 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.964 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.267218+0000", 2023-10-20T03:36:23.964 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00046247200000000001 2023-10-20T03:36:23.965 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.965 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.965 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.965 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.267257+0000", 2023-10-20T03:36:23.965 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8973000000000001e-05 2023-10-20T03:36:23.966 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.966 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.966 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.966 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.267276+0000", 2023-10-20T03:36:23.966 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8434999999999999e-05 2023-10-20T03:36:23.967 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.967 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.967 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.967 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.267440+0000", 2023-10-20T03:36:23.967 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000164081 2023-10-20T03:36:23.967 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.968 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.968 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.968 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.267467+0000", 2023-10-20T03:36:23.968 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7512e-05 2023-10-20T03:36:23.968 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.969 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.969 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.969 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.291988+0000", 2023-10-20T03:36:23.969 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.024520897 2023-10-20T03:36:23.969 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.970 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.970 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.970 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:18.292083+0000", 2023-10-20T03:36:23.970 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4573000000000004e-05 2023-10-20T03:36:23.970 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.970 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.971 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.971 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.971 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.971 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.1s4 43/31 ECSubWrite(tid=80, reqid=client.4570.0:306, at_version=43'14, trim_to=0'0, roll_forward_to=41'12))", 2023-10-20T03:36:23.971 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:25.329817+0000", 2023-10-20T03:36:23.972 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 58.626000961999999, 2023-10-20T03:36:23.972 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026006504999999999, 2023-10-20T03:36:23.972 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.972 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:23.972 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.329817+0000", 2023-10-20T03:36:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.973 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.974 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.974 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.329817+0000", 2023-10-20T03:36:23.974 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.977 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.977 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.977 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.978 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.329818+0000", 2023-10-20T03:36:23.978 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0699999999999997e-07 2023-10-20T03:36:23.978 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.978 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.978 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.979 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330431+0000", 2023-10-20T03:36:23.979 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00061258999999999999 2023-10-20T03:36:23.979 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.979 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.979 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.980 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330468+0000", 2023-10-20T03:36:23.980 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7651999999999998e-05 2023-10-20T03:36:23.980 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.980 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.980 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.980 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330493+0000", 2023-10-20T03:36:23.981 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4766e-05 2023-10-20T03:36:23.981 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.981 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.981 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.981 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330710+0000", 2023-10-20T03:36:23.982 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00021685000000000001 2023-10-20T03:36:23.982 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.982 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.982 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.982 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.330742+0000", 2023-10-20T03:36:23.983 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2406000000000001e-05 2023-10-20T03:36:23.983 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.983 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.983 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.983 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.355705+0000", 2023-10-20T03:36:23.984 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.024962682 2023-10-20T03:36:23.984 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.984 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.984 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.984 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:25.355824+0000", 2023-10-20T03:36:23.984 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000118952 2023-10-20T03:36:23.985 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.985 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.985 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.985 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.985 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.986 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s1 44/31 ECSubWrite(tid=18, reqid=client.4570.0:323, at_version=44'9, trim_to=0'0, roll_forward_to=34'7))", 2023-10-20T03:36:23.986 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.361569+0000", 2023-10-20T03:36:23.986 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 57.594249611000002, 2023-10-20T03:36:23.986 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019257209000000001, 2023-10-20T03:36:23.986 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.987 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:23.987 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.987 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.987 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.987 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361569+0000", 2023-10-20T03:36:23.987 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.988 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.988 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.988 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:23.988 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361569+0000", 2023-10-20T03:36:23.988 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.989 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.989 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.989 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:23.989 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361571+0000", 2023-10-20T03:36:23.989 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4370000000000001e-06 2023-10-20T03:36:23.990 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.990 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.990 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:23.990 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361886+0000", 2023-10-20T03:36:23.990 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00031429 2023-10-20T03:36:23.991 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.991 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.991 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:23.991 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361927+0000", 2023-10-20T03:36:23.991 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1174999999999998e-05 2023-10-20T03:36:23.992 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.992 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.992 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:23.992 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.361950+0000", 2023-10-20T03:36:23.992 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3048999999999999e-05 2023-10-20T03:36:23.993 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.993 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.993 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:23.993 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.362128+0000", 2023-10-20T03:36:23.993 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017819400000000001 2023-10-20T03:36:23.993 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.994 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.994 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:23.994 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.362154+0000", 2023-10-20T03:36:23.994 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6049000000000001e-05 2023-10-20T03:36:23.994 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.995 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.995 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:23.995 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.380718+0000", 2023-10-20T03:36:23.995 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.018564051000000002 2023-10-20T03:36:23.995 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.996 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.996 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:23.996 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.380826+0000", 2023-10-20T03:36:23.996 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000107964 2023-10-20T03:36:23.996 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.996 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:23.997 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:23.997 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:23.997 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.997 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s1 49/31 ECSubWrite(tid=39, reqid=client.4570.0:390, at_version=49'23, trim_to=0'0, roll_forward_to=47'21))", 2023-10-20T03:36:23.997 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:31.413927+0000", 2023-10-20T03:36:23.998 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 52.541891458999999, 2023-10-20T03:36:23.998 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026987964999999999, 2023-10-20T03:36:23.998 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:23.998 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:23.998 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:23.999 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:23.999 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:23.999 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.413927+0000", 2023-10-20T03:36:23.999 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:23.999 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.000 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.000 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.000 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.413927+0000", 2023-10-20T03:36:24.000 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.000 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.001 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.001 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.001 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.413928+0000", 2023-10-20T03:36:24.001 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5700000000000001e-07 2023-10-20T03:36:24.001 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.002 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.002 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.002 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.414488+0000", 2023-10-20T03:36:24.002 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00056035099999999995 2023-10-20T03:36:24.002 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.003 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.003 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.003 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.414523+0000", 2023-10-20T03:36:24.003 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5262000000000002e-05 2023-10-20T03:36:24.003 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.003 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.004 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.004 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.414546+0000", 2023-10-20T03:36:24.004 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2860000000000001e-05 2023-10-20T03:36:24.004 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.004 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.005 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.005 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.414713+0000", 2023-10-20T03:36:24.005 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016685199999999999 2023-10-20T03:36:24.005 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.005 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.006 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.006 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.414739+0000", 2023-10-20T03:36:24.006 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5721999999999999e-05 2023-10-20T03:36:24.006 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.006 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.007 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.007 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.440829+0000", 2023-10-20T03:36:24.007 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026089833999999999 2023-10-20T03:36:24.007 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.007 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.008 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.008 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:31.440915+0000", 2023-10-20T03:36:24.008 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6227e-05 2023-10-20T03:36:24.008 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.008 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.009 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.009 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.009 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.009 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s1 54/31 ECSubWrite(tid=65, reqid=client.4570.0:418, at_version=54'32, trim_to=0'0, roll_forward_to=51'31))", 2023-10-20T03:36:24.009 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:36.459265+0000", 2023-10-20T03:36:24.010 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 47.496553800999997, 2023-10-20T03:36:24.010 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016601141, 2023-10-20T03:36:24.010 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.010 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:24.010 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.011 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.011 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.011 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459265+0000", 2023-10-20T03:36:24.011 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.011 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.012 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.012 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.012 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459265+0000", 2023-10-20T03:36:24.012 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.012 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.012 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.013 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.013 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459265+0000", 2023-10-20T03:36:24.013 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3499999999999996e-07 2023-10-20T03:36:24.013 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.013 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.014 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.014 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459575+0000", 2023-10-20T03:36:24.014 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00030992299999999998 2023-10-20T03:36:24.014 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.014 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.015 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.015 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459606+0000", 2023-10-20T03:36:24.015 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0451e-05 2023-10-20T03:36:24.015 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.015 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.016 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.016 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459628+0000", 2023-10-20T03:36:24.016 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2248000000000001e-05 2023-10-20T03:36:24.016 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.016 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.016 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.017 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459783+0000", 2023-10-20T03:36:24.017 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015505099999999999 2023-10-20T03:36:24.017 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.017 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.017 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.018 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.459810+0000", 2023-10-20T03:36:24.018 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7061999999999999e-05 2023-10-20T03:36:24.018 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.018 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.018 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.019 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.475760+0000", 2023-10-20T03:36:24.019 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015950496000000002 2023-10-20T03:36:24.019 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.019 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.019 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.020 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:36.475866+0000", 2023-10-20T03:36:24.020 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000105375 2023-10-20T03:36:24.020 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.020 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.020 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.021 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.021 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.021 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.1s4 60/31 ECSubWrite(tid=190, reqid=client.4570.0:452, at_version=60'54, trim_to=0'0, roll_forward_to=58'48))", 2023-10-20T03:36:24.021 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.857727+0000", 2023-10-20T03:36:24.021 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 42.098090972000001, 2023-10-20T03:36:24.022 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016655356, 2023-10-20T03:36:24.022 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.022 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:24.022 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.022 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.023 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.023 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857727+0000", 2023-10-20T03:36:24.023 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.023 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.023 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.024 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.024 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857727+0000", 2023-10-20T03:36:24.024 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.024 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.024 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.024 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.025 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.857794+0000", 2023-10-20T03:36:24.025 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6687999999999996e-05 2023-10-20T03:36:24.025 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.025 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.025 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.026 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.858235+0000", 2023-10-20T03:36:24.026 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00044108999999999999 2023-10-20T03:36:24.026 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.026 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.026 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.027 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.858269+0000", 2023-10-20T03:36:24.027 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3828999999999997e-05 2023-10-20T03:36:24.027 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.027 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.027 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.028 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.858296+0000", 2023-10-20T03:36:24.028 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7243e-05 2023-10-20T03:36:24.028 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.028 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.028 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.029 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.858572+0000", 2023-10-20T03:36:24.029 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00027550399999999998 2023-10-20T03:36:24.029 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.029 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.029 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.030 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.858600+0000", 2023-10-20T03:36:24.030 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8436e-05 2023-10-20T03:36:24.030 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.030 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.030 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.031 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.874312+0000", 2023-10-20T03:36:24.031 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015711454 2023-10-20T03:36:24.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.031 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.031 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.032 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.874383+0000", 2023-10-20T03:36:24.032 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.1112000000000006e-05 2023-10-20T03:36:24.032 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.032 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.032 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.033 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.033 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.033 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.1s4 60/31 ECSubWrite(tid=195, reqid=client.4570.0:453, at_version=60'55, trim_to=0'0, roll_forward_to=60'52))", 2023-10-20T03:36:24.033 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.877727+0000", 2023-10-20T03:36:24.033 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 42.078091610000001, 2023-10-20T03:36:24.033 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.038516415999999998, 2023-10-20T03:36:24.034 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.034 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:24.034 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.034 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.034 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.034 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.877727+0000", 2023-10-20T03:36:24.035 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.035 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.035 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.035 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.035 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.877727+0000", 2023-10-20T03:36:24.035 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.036 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.036 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.877729+0000", 2023-10-20T03:36:24.036 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8500000000000001e-06 2023-10-20T03:36:24.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.037 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.037 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.037 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.878220+0000", 2023-10-20T03:36:24.037 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000490979 2023-10-20T03:36:24.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.038 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.038 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.038 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.878261+0000", 2023-10-20T03:36:24.038 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1088999999999999e-05 2023-10-20T03:36:24.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.039 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.039 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.039 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.878285+0000", 2023-10-20T03:36:24.039 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3839e-05 2023-10-20T03:36:24.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.040 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.040 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.040 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.878449+0000", 2023-10-20T03:36:24.040 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016397699999999999 2023-10-20T03:36:24.040 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.041 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.041 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.041 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.878475+0000", 2023-10-20T03:36:24.041 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6591999999999999e-05 2023-10-20T03:36:24.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.041 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.042 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.042 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.916178+0000", 2023-10-20T03:36:24.042 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.037702825000000002 2023-10-20T03:36:24.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.042 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.043 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.043 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.916243+0000", 2023-10-20T03:36:24.043 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5265000000000006e-05 2023-10-20T03:36:24.043 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.043 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.043 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.044 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.044 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.1s4 60/31 ECSubWrite(tid=199, reqid=client.4570.0:454, at_version=60'56, trim_to=0'0, roll_forward_to=60'52))", 2023-10-20T03:36:24.044 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.895555+0000", 2023-10-20T03:36:24.044 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 42.060263106000001, 2023-10-20T03:36:24.045 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.022269304, 2023-10-20T03:36:24.045 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.045 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:24.045 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.045 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.046 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.046 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.895555+0000", 2023-10-20T03:36:24.046 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.046 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.047 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.047 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.895555+0000", 2023-10-20T03:36:24.047 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.047 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.048 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.048 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.895557+0000", 2023-10-20T03:36:24.048 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6440000000000001e-06 2023-10-20T03:36:24.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.049 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.049 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.895676+0000", 2023-10-20T03:36:24.049 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011858600000000001 2023-10-20T03:36:24.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.049 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.049 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.050 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.895709+0000", 2023-10-20T03:36:24.050 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3176999999999998e-05 2023-10-20T03:36:24.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.050 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.050 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.051 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.895730+0000", 2023-10-20T03:36:24.051 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1313000000000001e-05 2023-10-20T03:36:24.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.051 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.051 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.052 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.903810+0000", 2023-10-20T03:36:24.052 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0080801840000000007 2023-10-20T03:36:24.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.052 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.052 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.053 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.903831+0000", 2023-10-20T03:36:24.053 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0990000000000001e-05 2023-10-20T03:36:24.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.053 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.053 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.053 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.917774+0000", 2023-10-20T03:36:24.054 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013943066 2023-10-20T03:36:24.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.054 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.054 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.054 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.917825+0000", 2023-10-20T03:36:24.055 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0343999999999998e-05 2023-10-20T03:36:24.055 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.055 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.055 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.056 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.056 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s1 69/31 ECSubWrite(tid=95, reqid=client.4570.0:552, at_version=69'41, trim_to=0'0, roll_forward_to=56'39))", 2023-10-20T03:36:24.056 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:50.954176+0000", 2023-10-20T03:36:24.056 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 33.001642081, 2023-10-20T03:36:24.056 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.031063285999999999, 2023-10-20T03:36:24.056 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.057 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:24.057 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.057 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.057 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.057 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.954176+0000", 2023-10-20T03:36:24.058 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.058 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.058 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.058 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.058 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.954176+0000", 2023-10-20T03:36:24.059 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.059 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.059 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.059 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.059 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.954177+0000", 2023-10-20T03:36:24.059 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1699999999999997e-07 2023-10-20T03:36:24.060 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.060 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.060 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.060 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.955059+0000", 2023-10-20T03:36:24.060 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00088211400000000003 2023-10-20T03:36:24.061 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.061 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.061 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.061 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.955105+0000", 2023-10-20T03:36:24.061 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5788999999999998e-05 2023-10-20T03:36:24.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.062 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.062 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.062 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.955133+0000", 2023-10-20T03:36:24.062 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7572e-05 2023-10-20T03:36:24.063 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.063 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.063 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.063 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.955339+0000", 2023-10-20T03:36:24.063 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000206467 2023-10-20T03:36:24.063 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.064 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.064 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.064 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.955357+0000", 2023-10-20T03:36:24.064 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7866000000000001e-05 2023-10-20T03:36:24.064 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.065 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.065 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.065 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.985166+0000", 2023-10-20T03:36:24.065 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029808845 2023-10-20T03:36:24.065 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.066 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.066 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.066 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:50.985240+0000", 2023-10-20T03:36:24.066 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3815999999999999e-05 2023-10-20T03:36:24.066 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.066 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.067 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.067 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.067 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.067 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:567 3.9s0 3:9d9c6ca9:::smithi17039300-24:head [append 0~671744] snapc 23=[23] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:36:24.067 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:52.036317+0000", 2023-10-20T03:36:24.068 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.919501138000001, 2023-10-20T03:36:24.068 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.043365328000000002, 2023-10-20T03:36:24.068 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.068 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.068 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.069 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.069 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.069 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 567 2023-10-20T03:36:24.069 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.069 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.070 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.070 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.070 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.036317+0000", 2023-10-20T03:36:24.070 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.070 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.070 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.071 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.071 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.036317+0000", 2023-10-20T03:36:24.071 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.071 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.071 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.072 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.072 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.036320+0000", 2023-10-20T03:36:24.072 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9440000000000001e-06 2023-10-20T03:36:24.072 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.072 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.073 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.073 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.037068+0000", 2023-10-20T03:36:24.073 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00074780599999999995 2023-10-20T03:36:24.073 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.073 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.074 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.074 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.037107+0000", 2023-10-20T03:36:24.074 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9305999999999999e-05 2023-10-20T03:36:24.074 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.074 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.074 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.075 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.037131+0000", 2023-10-20T03:36:24.075 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3297000000000001e-05 2023-10-20T03:36:24.075 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.075 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.075 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.076 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.037350+0000", 2023-10-20T03:36:24.076 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000218872 2023-10-20T03:36:24.076 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.076 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.076 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op must wait for map", 2023-10-20T03:36:24.076 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.037366+0000", 2023-10-20T03:36:24.077 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6407000000000001e-05 2023-10-20T03:36:24.077 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.077 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.077 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.077 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.043120+0000", 2023-10-20T03:36:24.077 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.005753831 2023-10-20T03:36:24.078 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.078 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.078 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.078 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.043262+0000", 2023-10-20T03:36:24.078 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014218 2023-10-20T03:36:24.078 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.079 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.079 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.079 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045535+0000", 2023-10-20T03:36:24.079 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0022726309999999998 2023-10-20T03:36:24.079 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.079 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.080 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.080 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.079562+0000", 2023-10-20T03:36:24.080 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.034027390999999997 2023-10-20T03:36:24.080 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.080 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.081 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.081 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.079613+0000", 2023-10-20T03:36:24.081 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1520999999999997e-05 2023-10-20T03:36:24.081 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.081 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.081 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.082 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.079683+0000", 2023-10-20T03:36:24.082 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9141999999999996e-05 2023-10-20T03:36:24.082 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.082 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.082 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.083 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.083 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.083 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:568 3.9s0 3:9d9c6ca9:::smithi17039300-24:head [append 0~753664] snapc 23=[23] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:36:24.083 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:52.042436+0000", 2023-10-20T03:36:24.083 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.913382085999999, 2023-10-20T03:36:24.083 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.048945191999999998, 2023-10-20T03:36:24.084 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.084 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.084 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.084 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.084 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.085 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 568 2023-10-20T03:36:24.085 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.085 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.085 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.085 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.086 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.042436+0000", 2023-10-20T03:36:24.086 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.086 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.086 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.086 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.087 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.042436+0000", 2023-10-20T03:36:24.087 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.087 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.087 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.087 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.088 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.042439+0000", 2023-10-20T03:36:24.088 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9189999999999999e-06 2023-10-20T03:36:24.088 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.088 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.088 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.089 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.043409+0000", 2023-10-20T03:36:24.089 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00097019599999999999 2023-10-20T03:36:24.089 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.089 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.089 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.090 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.043442+0000", 2023-10-20T03:36:24.090 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2407000000000003e-05 2023-10-20T03:36:24.090 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.090 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.090 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.090 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.043464+0000", 2023-10-20T03:36:24.091 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2218999999999999e-05 2023-10-20T03:36:24.091 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.091 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.091 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.091 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.064466+0000", 2023-10-20T03:36:24.092 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.021001891000000002 2023-10-20T03:36:24.092 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.092 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.092 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.092 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.064576+0000", 2023-10-20T03:36:24.092 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110419 2023-10-20T03:36:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.065682+0000", 2023-10-20T03:36:24.093 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0011051450000000001 2023-10-20T03:36:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.079724+0000", 2023-10-20T03:36:24.094 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014042532999999999 2023-10-20T03:36:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.091307+0000", 2023-10-20T03:36:24.095 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011582535 2023-10-20T03:36:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.091382+0000", 2023-10-20T03:36:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4927999999999998e-05 2023-10-20T03:36:24.096 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.097 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:569 3.9s0 3:9d9c6ca9:::smithi17039300-24:head [append 0~409600] snapc 23=[23] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:36:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:52.044686+0000", 2023-10-20T03:36:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.911132380000002, 2023-10-20T03:36:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.046937062000000002, 2023-10-20T03:36:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.098 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.099 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 569 2023-10-20T03:36:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.044686+0000", 2023-10-20T03:36:24.100 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.044686+0000", 2023-10-20T03:36:24.101 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.044687+0000", 2023-10-20T03:36:24.102 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1960000000000001e-06 2023-10-20T03:36:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.044997+0000", 2023-10-20T03:36:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00031009199999999998 2023-10-20T03:36:24.103 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.104 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.104 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.104 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045009+0000", 2023-10-20T03:36:24.104 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1197e-05 2023-10-20T03:36:24.104 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.105 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.105 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.105 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045015+0000", 2023-10-20T03:36:24.105 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.7129999999999999e-06 2023-10-20T03:36:24.105 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.105 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.106 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.106 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.073854+0000", 2023-10-20T03:36:24.106 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.02883908 2023-10-20T03:36:24.106 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.106 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.107 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.107 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.074058+0000", 2023-10-20T03:36:24.107 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020349100000000001 2023-10-20T03:36:24.107 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.107 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.108 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.108 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.075191+0000", 2023-10-20T03:36:24.108 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0011331570000000001 2023-10-20T03:36:24.108 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.108 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.109 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.109 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.080702+0000", 2023-10-20T03:36:24.109 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0055109879999999997 2023-10-20T03:36:24.109 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.109 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.109 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.110 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.091534+0000", 2023-10-20T03:36:24.110 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010832329999999999 2023-10-20T03:36:24.110 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.110 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.110 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.111 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.091623+0000", 2023-10-20T03:36:24.111 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.8818000000000002e-05 2023-10-20T03:36:24.111 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.111 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.111 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.112 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.112 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.112 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:570 3.9s0 3:9d9c6ca9:::smithi17039300-24:head [setxattr _header (57) in=7b] snapc 23=[23] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:36:24.112 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:52.045029+0000", 2023-10-20T03:36:24.112 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.910789552000001, 2023-10-20T03:36:24.113 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.047568815, 2023-10-20T03:36:24.113 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.113 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.113 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.113 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.114 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.114 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 570 2023-10-20T03:36:24.114 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.114 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.114 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.115 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.115 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045029+0000", 2023-10-20T03:36:24.115 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.115 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.115 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.115 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.116 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045029+0000", 2023-10-20T03:36:24.116 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.116 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.116 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.116 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.116 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045029+0000", 2023-10-20T03:36:24.117 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.46e-07 2023-10-20T03:36:24.117 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.117 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.117 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.117 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045033+0000", 2023-10-20T03:36:24.118 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6430000000000001e-06 2023-10-20T03:36:24.118 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.118 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.118 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.118 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045038+0000", 2023-10-20T03:36:24.118 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6480000000000002e-06 2023-10-20T03:36:24.119 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.119 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.119 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.119 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045044+0000", 2023-10-20T03:36:24.119 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9510000000000003e-06 2023-10-20T03:36:24.120 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.120 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.120 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.120 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.079942+0000", 2023-10-20T03:36:24.120 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.034898339 2023-10-20T03:36:24.121 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.121 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.121 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.121 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.080006+0000", 2023-10-20T03:36:24.121 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4095000000000002e-05 2023-10-20T03:36:24.122 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.122 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.122 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.122 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.080303+0000", 2023-10-20T03:36:24.122 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00029723299999999999 2023-10-20T03:36:24.123 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.123 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.123 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.123 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.081056+0000", 2023-10-20T03:36:24.123 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00075240799999999998 2023-10-20T03:36:24.124 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.124 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.124 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.124 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.092497+0000", 2023-10-20T03:36:24.124 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011441046 2023-10-20T03:36:24.125 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.125 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.125 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.125 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.092598+0000", 2023-10-20T03:36:24.125 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101006 2023-10-20T03:36:24.125 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.126 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.126 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.126 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.126 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.126 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:571 3.9s0 3:9d9c6ca9:::smithi17039300-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2023-10-20T03:36:24.127 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:52.045049+0000", 2023-10-20T03:36:24.127 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.910769269999999, 2023-10-20T03:36:24.127 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.050729387000000001, 2023-10-20T03:36:24.127 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.127 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:24.128 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.128 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.128 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.128 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 571 2023-10-20T03:36:24.128 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.128 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.129 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.129 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.129 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045049+0000", 2023-10-20T03:36:24.129 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.129 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.130 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.130 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.130 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045049+0000", 2023-10-20T03:36:24.130 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.131 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.131 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.131 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045049+0000", 2023-10-20T03:36:24.131 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.84e-07 2023-10-20T03:36:24.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.132 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.132 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.132 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045051+0000", 2023-10-20T03:36:24.132 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0080000000000001e-06 2023-10-20T03:36:24.132 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.133 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.133 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.133 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045056+0000", 2023-10-20T03:36:24.133 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1799999999999998e-06 2023-10-20T03:36:24.133 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.133 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.134 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.134 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.045059+0000", 2023-10-20T03:36:24.134 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4020000000000002e-06 2023-10-20T03:36:24.134 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.134 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.135 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.135 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.080592+0000", 2023-10-20T03:36:24.135 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.035532546999999998 2023-10-20T03:36:24.135 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.135 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.136 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:24.136 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.080692+0000", 2023-10-20T03:36:24.136 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000100145 2023-10-20T03:36:24.136 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.136 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.136 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.137 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.093194+0000", 2023-10-20T03:36:24.137 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012502704999999999 2023-10-20T03:36:24.137 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.137 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.137 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.138 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.093358+0000", 2023-10-20T03:36:24.138 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016385400000000001 2023-10-20T03:36:24.138 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.138 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.138 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.138 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:52.095779+0000", 2023-10-20T03:36:24.139 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0024202619999999998 2023-10-20T03:36:24.139 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.139 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.139 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.139 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.140 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.140 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.2s6 74/31 ECSubWrite(tid=65, reqid=client.4570.0:615, at_version=74'48, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:24.140 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:56.031451+0000", 2023-10-20T03:36:24.140 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 27.924367782000001, 2023-10-20T03:36:24.140 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.032107326999999998, 2023-10-20T03:36:24.140 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.141 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:24.141 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.141 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.141 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.141 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:56.031451+0000", 2023-10-20T03:36:24.142 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.142 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.142 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.142 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.142 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:56.031451+0000", 2023-10-20T03:36:24.143 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.143 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.143 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.143 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.143 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:56.031452+0000", 2023-10-20T03:36:24.144 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.42e-06 2023-10-20T03:36:24.144 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.144 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.144 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.144 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:56.032197+0000", 2023-10-20T03:36:24.145 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000744614 2023-10-20T03:36:24.145 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.145 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.145 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.145 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:56.032243+0000", 2023-10-20T03:36:24.145 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6637000000000001e-05 2023-10-20T03:36:24.146 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.146 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.146 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.146 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:56.032268+0000", 2023-10-20T03:36:24.146 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4868e-05 2023-10-20T03:36:24.147 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.147 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.147 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.147 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:56.032385+0000", 2023-10-20T03:36:24.147 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011717 2023-10-20T03:36:24.148 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.148 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.148 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.148 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:56.032419+0000", 2023-10-20T03:36:24.148 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3315000000000002e-05 2023-10-20T03:36:24.148 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.149 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.149 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.149 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:56.063460+0000", 2023-10-20T03:36:24.149 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.031041144999999999 2023-10-20T03:36:24.149 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.150 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.150 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.150 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:56.063558+0000", 2023-10-20T03:36:24.150 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.8158000000000004e-05 2023-10-20T03:36:24.150 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.151 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.151 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.151 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.151 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.151 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.0s1 75/31 ECSubWrite(tid=97, reqid=client.4570.0:628, at_version=75'43, trim_to=0'0, roll_forward_to=69'41))", 2023-10-20T03:36:24.151 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:57.036116+0000", 2023-10-20T03:36:24.152 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 26.919702389000001, 2023-10-20T03:36:24.152 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.036822700999999999, 2023-10-20T03:36:24.152 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.152 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:24.152 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.153 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.153 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.153 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.036116+0000", 2023-10-20T03:36:24.153 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.153 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.153 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.154 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.154 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.036116+0000", 2023-10-20T03:36:24.154 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.154 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.154 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.155 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.155 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.036117+0000", 2023-10-20T03:36:24.155 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.153e-06 2023-10-20T03:36:24.155 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.155 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.156 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.156 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.036963+0000", 2023-10-20T03:36:24.156 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00084562299999999999 2023-10-20T03:36:24.156 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.156 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.156 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.157 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.037011+0000", 2023-10-20T03:36:24.157 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7753999999999997e-05 2023-10-20T03:36:24.157 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.157 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.157 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.158 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.037036+0000", 2023-10-20T03:36:24.158 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5519000000000001e-05 2023-10-20T03:36:24.158 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.158 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.158 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.159 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.037263+0000", 2023-10-20T03:36:24.159 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000227147 2023-10-20T03:36:24.159 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.159 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.159 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.160 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.037288+0000", 2023-10-20T03:36:24.160 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4799000000000001e-05 2023-10-20T03:36:24.160 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.160 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.160 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.161 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.072838+0000", 2023-10-20T03:36:24.161 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.035549732000000001 2023-10-20T03:36:24.161 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.161 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.161 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.162 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:57.072939+0000", 2023-10-20T03:36:24.162 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000100974 2023-10-20T03:36:24.162 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.162 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.162 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.162 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.163 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.163 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:821 3.9s0 3:97c0e862:::smithi17039300-18:head [delete] snapc 3b=[39,38,37] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95)", 2023-10-20T03:36:24.163 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:36:17.175482+0000", 2023-10-20T03:36:24.163 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 6.780336095, 2023-10-20T03:36:24.163 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.018727297, 2023-10-20T03:36:24.163 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.164 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:24.164 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.164 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.164 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.165 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 821 2023-10-20T03:36:24.165 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.165 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.165 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.165 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.165 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:17.175482+0000", 2023-10-20T03:36:24.166 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.166 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.166 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.166 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.166 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:17.175482+0000", 2023-10-20T03:36:24.167 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.167 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.167 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.167 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.167 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:17.175490+0000", 2023-10-20T03:36:24.168 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9920000000000007e-06 2023-10-20T03:36:24.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.168 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.168 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.168 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:17.175498+0000", 2023-10-20T03:36:24.169 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.503e-06 2023-10-20T03:36:24.169 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.169 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.169 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.169 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:17.175511+0000", 2023-10-20T03:36:24.170 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3269e-05 2023-10-20T03:36:24.170 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.170 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.170 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.170 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:17.175523+0000", 2023-10-20T03:36:24.171 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1483e-05 2023-10-20T03:36:24.171 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.171 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.171 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.171 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:17.175921+0000", 2023-10-20T03:36:24.171 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000398374 2023-10-20T03:36:24.172 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.172 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.172 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2023-10-20T03:36:24.172 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:17.175926+0000", 2023-10-20T03:36:24.172 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9810000000000003e-06 2023-10-20T03:36:24.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.173 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.173 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.173 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:17.187902+0000", 2023-10-20T03:36:24.173 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011976393 2023-10-20T03:36:24.174 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.174 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.174 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.174 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:17.188544+0000", 2023-10-20T03:36:24.174 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00064185900000000005 2023-10-20T03:36:24.175 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.175 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.175 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.175 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:17.194210+0000", 2023-10-20T03:36:24.175 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.005665443 2023-10-20T03:36:24.176 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.176 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.176 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.176 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.176 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.177 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.2s6 96/31 ECSubWrite(tid=102, reqid=client.4570.0:824, at_version=96'81, trim_to=0'0, roll_forward_to=96'79))", 2023-10-20T03:36:24.177 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:36:18.224478+0000", 2023-10-20T03:36:24.177 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 5.7313399990000002, 2023-10-20T03:36:24.177 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019103835999999999, 2023-10-20T03:36:24.177 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.177 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:24.178 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.178 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.178 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.178 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:18.224478+0000", 2023-10-20T03:36:24.178 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.179 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.179 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.179 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.179 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:18.224478+0000", 2023-10-20T03:36:24.179 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.180 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.180 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.180 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.180 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:18.224479+0000", 2023-10-20T03:36:24.180 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7300000000000005e-07 2023-10-20T03:36:24.180 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.181 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.181 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.181 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:18.224990+0000", 2023-10-20T03:36:24.181 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00051062299999999998 2023-10-20T03:36:24.181 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.182 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.182 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.182 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:18.225003+0000", 2023-10-20T03:36:24.182 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3091000000000001e-05 2023-10-20T03:36:24.182 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.182 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.183 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.183 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:18.225012+0000", 2023-10-20T03:36:24.183 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3559999999999999e-06 2023-10-20T03:36:24.183 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.183 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.184 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.184 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:18.225185+0000", 2023-10-20T03:36:24.184 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017228999999999999 2023-10-20T03:36:24.184 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.184 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.185 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.185 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:18.225223+0000", 2023-10-20T03:36:24.185 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8512e-05 2023-10-20T03:36:24.185 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.185 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.185 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.186 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:18.243479+0000", 2023-10-20T03:36:24.186 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.018255835000000002 2023-10-20T03:36:24.186 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.186 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.186 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.187 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:18.243582+0000", 2023-10-20T03:36:24.187 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000103256 2023-10-20T03:36:24.187 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.187 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.187 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.188 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.188 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:885 3.9s0 3:97c0e862:::smithi17039300-18:head [copy-from ver 11] snapc 42=[42] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2023-10-20T03:36:24.188 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:36:23.239482+0000", 2023-10-20T03:36:24.188 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.71633618899999996, 2023-10-20T03:36:24.189 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.062503498000000005, 2023-10-20T03:36:24.189 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.189 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.189 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.189 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.190 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.190 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 885 2023-10-20T03:36:24.190 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.190 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.190 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.191 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.191 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239482+0000", 2023-10-20T03:36:24.191 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.191 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.192 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.192 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239482+0000", 2023-10-20T03:36:24.192 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.192 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.192 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.192 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.193 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239484+0000", 2023-10-20T03:36:24.193 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5629999999999999e-06 2023-10-20T03:36:24.193 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.193 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.193 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.194 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239492+0000", 2023-10-20T03:36:24.194 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1580000000000005e-06 2023-10-20T03:36:24.194 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.194 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.194 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.195 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239508+0000", 2023-10-20T03:36:24.195 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5940999999999999e-05 2023-10-20T03:36:24.195 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.195 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.195 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.196 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239517+0000", 2023-10-20T03:36:24.196 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9439999999999994e-06 2023-10-20T03:36:24.196 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.196 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.196 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.197 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239575+0000", 2023-10-20T03:36:24.197 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7840999999999997e-05 2023-10-20T03:36:24.197 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.197 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.197 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.197 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239653+0000", 2023-10-20T03:36:24.198 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8720999999999994e-05 2023-10-20T03:36:24.198 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.198 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.198 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.198 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.253670+0000", 2023-10-20T03:36:24.199 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014016868 2023-10-20T03:36:24.199 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.199 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.199 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.199 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.272630+0000", 2023-10-20T03:36:24.200 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.018959888000000001 2023-10-20T03:36:24.200 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.200 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.200 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.200 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.301598+0000", 2023-10-20T03:36:24.201 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.028967577000000001 2023-10-20T03:36:24.201 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.201 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.201 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.201 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.301986+0000", 2023-10-20T03:36:24.201 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00038799700000000001 2023-10-20T03:36:24.202 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.202 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.202 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.202 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.202 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.203 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:886 3.9s0 3:97c0e862:::smithi17039300-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e101)", 2023-10-20T03:36:24.203 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:36:23.239534+0000", 2023-10-20T03:36:24.203 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.71628409199999998, 2023-10-20T03:36:24.203 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.033037996, 2023-10-20T03:36:24.203 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.203 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:24.204 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.204 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.204 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.204 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 886 2023-10-20T03:36:24.204 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.205 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.205 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.205 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.205 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239534+0000", 2023-10-20T03:36:24.205 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.205 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.206 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.206 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.206 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239534+0000", 2023-10-20T03:36:24.206 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.206 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.207 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.207 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239535+0000", 2023-10-20T03:36:24.207 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8500000000000002e-07 2023-10-20T03:36:24.207 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.207 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.208 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.208 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239537+0000", 2023-10-20T03:36:24.208 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4899999999999999e-06 2023-10-20T03:36:24.208 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.208 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.208 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.209 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239545+0000", 2023-10-20T03:36:24.209 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.174e-06 2023-10-20T03:36:24.209 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.209 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.209 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.210 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239550+0000", 2023-10-20T03:36:24.210 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2710000000000003e-06 2023-10-20T03:36:24.210 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.210 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.210 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.210 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239811+0000", 2023-10-20T03:36:24.211 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000261661 2023-10-20T03:36:24.211 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.211 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.211 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for blocked object", 2023-10-20T03:36:24.211 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.239910+0000", 2023-10-20T03:36:24.212 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.8743e-05 2023-10-20T03:36:24.212 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.212 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.212 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.212 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.271957+0000", 2023-10-20T03:36:24.212 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.032046850000000002 2023-10-20T03:36:24.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.213 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.213 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.213 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.272132+0000", 2023-10-20T03:36:24.213 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017497500000000001 2023-10-20T03:36:24.214 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.214 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.214 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.214 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:23.272572+0000", 2023-10-20T03:36:24.214 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00044044700000000001 2023-10-20T03:36:24.215 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.215 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.215 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.215 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.215 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.216 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:24.217 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-10-20T03:36:24.218 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:24.216+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:24.244 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is set 2023-10-20T03:36:24.253 INFO:tasks.rados.rados.0.smithi170.stdout:373: done (3 left) 2023-10-20T03:36:24.253 INFO:tasks.rados.rados.0.smithi170.stdout:374: done (2 left) 2023-10-20T03:36:24.254 INFO:tasks.rados.rados.0.smithi170.stdout:375: done (1 left) 2023-10-20T03:36:24.254 INFO:tasks.rados.rados.0.smithi170.stdout:376: done (0 left) 2023-10-20T03:36:24.254 INFO:tasks.rados.rados.0.smithi170.stdout:377: copy_from oid 4 from oid 25 current snap is 34 2023-10-20T03:36:24.254 INFO:tasks.rados.rados.0.smithi170.stdout:378: read oid 49 snap -1 2023-10-20T03:36:24.254 INFO:tasks.rados.rados.0.smithi170.stdout:378: expect (ObjNum 91 snap 11 seq_num 91) 2023-10-20T03:36:24.255 INFO:tasks.rados.rados.0.smithi170.stdout:379: rollback oid 50 current snap is 34 2023-10-20T03:36:24.255 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 50 to 33 2023-10-20T03:36:24.255 INFO:tasks.rados.rados.0.smithi170.stdout:380: rollback oid 28 current snap is 34 2023-10-20T03:36:24.255 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 28 to 33 2023-10-20T03:36:24.255 INFO:tasks.rados.rados.0.smithi170.stdout:381: read oid 31 snap -1 2023-10-20T03:36:24.256 INFO:tasks.rados.rados.0.smithi170.stdout:381: expect (ObjNum 78 snap 7 seq_num 78) 2023-10-20T03:36:24.256 INFO:tasks.rados.rados.0.smithi170.stdout:382: rmattr oid 9 current snap is 34 2023-10-20T03:36:24.256 INFO:tasks.rados.rados.0.smithi170.stdout:382: done (5 left) 2023-10-20T03:36:24.256 INFO:tasks.rados.rados.0.smithi170.stdout:383: rollback oid 35 current snap is 34 2023-10-20T03:36:24.256 INFO:tasks.rados.rados.0.smithi170.stdout:rollback oid 35 to 33 2023-10-20T03:36:24.256 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 12 current snap is 34 2023-10-20T03:36:24.257 INFO:tasks.rados.rados.0.smithi170.stdout:384: seq_num 153 ranges {0=163840} 2023-10-20T03:36:24.258 INFO:tasks.rados.rados.0.smithi170.stdout:384: writing smithi17039300-12 from 0 to 163840 tid 1 2023-10-20T03:36:24.258 INFO:tasks.rados.rados.0.smithi170.stdout:385: read oid 46 snap -1 2023-10-20T03:36:24.258 INFO:tasks.rados.rados.0.smithi170.stdout:385: expect (ObjNum 105 snap 16 seq_num 105) 2023-10-20T03:36:24.259 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 43 current snap is 34 2023-10-20T03:36:24.259 INFO:tasks.rados.rados.0.smithi170.stdout:386: seq_num 154 ranges {802816=180224} 2023-10-20T03:36:24.263 INFO:tasks.rados.rados.0.smithi170.stdout:386: writing smithi17039300-43 from 802816 to 983040 tid 1 2023-10-20T03:36:24.264 INFO:tasks.rados.rados.0.smithi170.stdout:379: finishing rollback tid 1 to smithi17039300-50 2023-10-20T03:36:24.264 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 50 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2023-10-20T03:36:24.264 INFO:tasks.rados.rados.0.smithi170.stdout:379: done (8 left) 2023-10-20T03:36:24.265 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 6 current snap is 34 2023-10-20T03:36:24.265 INFO:tasks.rados.rados.0.smithi170.stdout:387: seq_num 155 ranges {0=212992} 2023-10-20T03:36:24.266 INFO:tasks.rados.rados.0.smithi170.stdout:387: writing smithi17039300-6 from 0 to 212992 tid 1 2023-10-20T03:36:24.266 INFO:tasks.rados.rados.0.smithi170.stdout:388: setattr oid 22 current snap is 34 2023-10-20T03:36:24.277 INFO:tasks.rados.rados.0.smithi170.stdout:380: finishing rollback tid 0 to smithi17039300-28 2023-10-20T03:36:24.279 INFO:tasks.rados.rados.0.smithi170.stdout:389: snap_remove snap 32 2023-10-20T03:36:24.317 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:24.312+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:24.390 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:24.390 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:36:24.390 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2023-10-20T03:36:24.390 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:24.402 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-10-20T03:36:24.419 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:24.415+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:24.468 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:36:24.520 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:36:24.516+0000 7fa8564c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:36:24.564 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:24.564 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2023-10-20T03:36:24.565 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2023-10-20T03:36:24.565 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2023-10-20T03:36:24.565 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:24.578 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-10-20T03:36:24.624 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:24.620+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:24.627 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:36:24.627 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:36:24.647 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:36:24.648 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:36:24.669 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:36:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:36:24.690 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:36:24.691 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:36:24.701 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2023-10-20T03:36:24.701 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2023-10-20T03:36:24.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2023-10-20T03:36:24.702 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2023-10-20T03:36:24.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.702 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:208 3.es0 3:7dffa048:::smithi17039300-5:head [append 0~688128] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:36:24.702 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.269565+0000", 2023-10-20T03:36:24.703 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 65.428545193000005, 2023-10-20T03:36:24.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029923571, 2023-10-20T03:36:24.703 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.703 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.703 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.704 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.704 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.704 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 208 2023-10-20T03:36:24.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.704 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.269565+0000", 2023-10-20T03:36:24.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.269565+0000", 2023-10-20T03:36:24.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.707 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.707 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.707 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.269569+0000", 2023-10-20T03:36:24.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0230000000000001e-06 2023-10-20T03:36:24.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.708 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.708 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270337+0000", 2023-10-20T03:36:24.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00076892900000000001 2023-10-20T03:36:24.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270372+0000", 2023-10-20T03:36:24.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4652e-05 2023-10-20T03:36:24.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270391+0000", 2023-10-20T03:36:24.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8819000000000001e-05 2023-10-20T03:36:24.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270555+0000", 2023-10-20T03:36:24.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016383700000000001 2023-10-20T03:36:24.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.270730+0000", 2023-10-20T03:36:24.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017536 2023-10-20T03:36:24.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.713 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.713 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.272967+0000", 2023-10-20T03:36:24.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002236601 2023-10-20T03:36:24.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.298706+0000", 2023-10-20T03:36:24.715 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025739482000000001 2023-10-20T03:36:24.715 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.715 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.715 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.715 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299042+0000", 2023-10-20T03:36:24.716 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00033608100000000001 2023-10-20T03:36:24.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.716 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.716 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.716 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.299489+0000", 2023-10-20T03:36:24.717 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00044678699999999998 2023-10-20T03:36:24.717 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.717 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.717 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.718 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.718 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:210 3.es0 3:7dffa048:::smithi17039300-5:head [setxattr _header (55) in=7b] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:36:24.718 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.278490+0000", 2023-10-20T03:36:24.718 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 65.419620480000006, 2023-10-20T03:36:24.718 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027549357999999999, 2023-10-20T03:36:24.718 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.719 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.719 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.719 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.719 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.719 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210 2023-10-20T03:36:24.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278490+0000", 2023-10-20T03:36:24.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278490+0000", 2023-10-20T03:36:24.721 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278492+0000", 2023-10-20T03:36:24.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.59e-06 2023-10-20T03:36:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278501+0000", 2023-10-20T03:36:24.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.986e-06 2023-10-20T03:36:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278519+0000", 2023-10-20T03:36:24.724 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8238999999999999e-05 2023-10-20T03:36:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278536+0000", 2023-10-20T03:36:24.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7376000000000001e-05 2023-10-20T03:36:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.301212+0000", 2023-10-20T03:36:24.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.022675563999999999 2023-10-20T03:36:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.301368+0000", 2023-10-20T03:36:24.727 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015616 2023-10-20T03:36:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.302117+0000", 2023-10-20T03:36:24.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00074924300000000002 2023-10-20T03:36:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.303929+0000", 2023-10-20T03:36:24.729 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0018115830000000001 2023-10-20T03:36:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.305714+0000", 2023-10-20T03:36:24.730 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001785502 2023-10-20T03:36:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.306040+0000", 2023-10-20T03:36:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00032511500000000001 2023-10-20T03:36:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:211 3.es0 3:7dffa048:::smithi17039300-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-10-20T03:36:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:19.278563+0000", 2023-10-20T03:36:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 65.419548172999995, 2023-10-20T03:36:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030205164, 2023-10-20T03:36:24.733 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.734 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:24.734 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.734 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.734 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.735 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211 2023-10-20T03:36:24.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.735 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.735 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.735 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278563+0000", 2023-10-20T03:36:24.736 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.736 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.736 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.736 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278563+0000", 2023-10-20T03:36:24.736 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.737 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.737 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.737 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278564+0000", 2023-10-20T03:36:24.738 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.826e-06 2023-10-20T03:36:24.738 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.738 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.738 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.738 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278572+0000", 2023-10-20T03:36:24.739 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0299999999999994e-06 2023-10-20T03:36:24.739 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.739 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.739 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.739 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278598+0000", 2023-10-20T03:36:24.739 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5593e-05 2023-10-20T03:36:24.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.740 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.740 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.740 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.278619+0000", 2023-10-20T03:36:24.740 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0755e-05 2023-10-20T03:36:24.741 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.741 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.741 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.302759+0000", 2023-10-20T03:36:24.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.024140663 2023-10-20T03:36:24.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:24.742 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.303005+0000", 2023-10-20T03:36:24.742 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024529499999999997 2023-10-20T03:36:24.743 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.743 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.743 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.743 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.306691+0000", 2023-10-20T03:36:24.743 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0036863809999999999 2023-10-20T03:36:24.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.744 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.744 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.744 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.306820+0000", 2023-10-20T03:36:24.744 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012892200000000001 2023-10-20T03:36:24.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.745 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.745 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:19.308768+0000", 2023-10-20T03:36:24.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001947699 2023-10-20T03:36:24.745 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.746 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.746 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.746 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.746 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.ds2 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:24.747 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:26.388826+0000", 2023-10-20T03:36:24.747 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 58.309285084000003, 2023-10-20T03:36:24.747 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027218437000000002, 2023-10-20T03:36:24.747 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.747 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:24.747 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.748 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.748 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.748 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388826+0000", 2023-10-20T03:36:24.748 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.748 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.749 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.749 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.749 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388826+0000", 2023-10-20T03:36:24.749 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.750 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.750 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.750 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.388827+0000", 2023-10-20T03:36:24.750 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5300000000000002e-07 2023-10-20T03:36:24.751 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.751 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.751 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.751 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389257+0000", 2023-10-20T03:36:24.751 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00043021500000000001 2023-10-20T03:36:24.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.752 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.752 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389281+0000", 2023-10-20T03:36:24.752 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4697000000000001e-05 2023-10-20T03:36:24.753 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.753 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.753 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.753 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389295+0000", 2023-10-20T03:36:24.753 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3491e-05 2023-10-20T03:36:24.754 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.754 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.754 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.754 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389419+0000", 2023-10-20T03:36:24.754 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012411400000000001 2023-10-20T03:36:24.755 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.755 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.755 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.755 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.389432+0000", 2023-10-20T03:36:24.755 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2551000000000001e-05 2023-10-20T03:36:24.756 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.756 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.756 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.756 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.415986+0000", 2023-10-20T03:36:24.756 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026554069999999999 2023-10-20T03:36:24.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.757 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.757 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.757 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:26.416044+0000", 2023-10-20T03:36:24.757 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8346e-05 2023-10-20T03:36:24.758 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.758 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.758 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.759 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.ds2 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:36:24.759 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:30.392840+0000", 2023-10-20T03:36:24.759 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 54.305270878000002, 2023-10-20T03:36:24.759 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.037253397000000001, 2023-10-20T03:36:24.759 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.759 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:24.760 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.760 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.760 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.760 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.392840+0000", 2023-10-20T03:36:24.760 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.761 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.761 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.761 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.392840+0000", 2023-10-20T03:36:24.761 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.762 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.762 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.762 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.392841+0000", 2023-10-20T03:36:24.762 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.567e-06 2023-10-20T03:36:24.763 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.763 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.763 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.763 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.394858+0000", 2023-10-20T03:36:24.763 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020165589999999998 2023-10-20T03:36:24.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.764 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.764 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.764 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.394892+0000", 2023-10-20T03:36:24.764 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4001999999999997e-05 2023-10-20T03:36:24.765 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.765 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.765 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.765 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.394916+0000", 2023-10-20T03:36:24.765 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4002000000000002e-05 2023-10-20T03:36:24.766 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.766 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.766 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.766 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.395085+0000", 2023-10-20T03:36:24.766 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016879899999999999 2023-10-20T03:36:24.767 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.767 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.767 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.767 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.395112+0000", 2023-10-20T03:36:24.767 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7032000000000001e-05 2023-10-20T03:36:24.768 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.768 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.768 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.768 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.430024+0000", 2023-10-20T03:36:24.768 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.034911958 2023-10-20T03:36:24.769 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.769 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.769 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.769 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:30.430093+0000", 2023-10-20T03:36:24.769 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9478000000000001e-05 2023-10-20T03:36:24.769 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.770 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.770 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.770 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.770 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.fs5 53/47 ECSubWrite(tid=57, reqid=client.4570.0:408, at_version=53'31, trim_to=0'0, roll_forward_to=53'29))", 2023-10-20T03:36:24.771 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:35.452469+0000", 2023-10-20T03:36:24.771 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 49.245641327000001, 2023-10-20T03:36:24.771 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.033778967999999999, 2023-10-20T03:36:24.771 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.771 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452469+0000", 2023-10-20T03:36:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.773 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.773 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.773 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452469+0000", 2023-10-20T03:36:24.773 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.774 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.774 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.774 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452470+0000", 2023-10-20T03:36:24.774 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6299999999999993e-07 2023-10-20T03:36:24.774 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.775 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.775 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.775 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452906+0000", 2023-10-20T03:36:24.775 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00043586900000000002 2023-10-20T03:36:24.775 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.776 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.776 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.776 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452933+0000", 2023-10-20T03:36:24.776 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.69e-05 2023-10-20T03:36:24.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.777 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.777 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.452952+0000", 2023-10-20T03:36:24.777 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9089e-05 2023-10-20T03:36:24.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.778 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.778 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.778 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.453188+0000", 2023-10-20T03:36:24.778 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00023577799999999999 2023-10-20T03:36:24.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.779 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.779 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.779 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.453218+0000", 2023-10-20T03:36:24.779 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0442000000000001e-05 2023-10-20T03:36:24.779 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.780 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.780 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.780 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.486138+0000", 2023-10-20T03:36:24.780 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.032919118999999997 2023-10-20T03:36:24.780 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.781 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.781 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.781 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:35.486248+0000", 2023-10-20T03:36:24.781 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110808 2023-10-20T03:36:24.781 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.782 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.782 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.782 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.782 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.782 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "MOSDECSubOpWrite(3.7s1 60/31 ECSubWrite(tid=198, reqid=client.4570.0:458, at_version=60'8, trim_to=0'0, roll_forward_to=38'7))", 2023-10-20T03:36:24.783 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:41.888718+0000", 2023-10-20T03:36:24.783 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 42.809392979000002, 2023-10-20T03:36:24.783 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.027011496999999999, 2023-10-20T03:36:24.783 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.783 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2023-10-20T03:36:24.784 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.784 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.784 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.784 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888718+0000", 2023-10-20T03:36:24.784 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.785 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.785 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.785 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.785 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888718+0000", 2023-10-20T03:36:24.785 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.785 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.786 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.786 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.786 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.888719+0000", 2023-10-20T03:36:24.786 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7230000000000001e-06 2023-10-20T03:36:24.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.787 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.787 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.787 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889432+0000", 2023-10-20T03:36:24.787 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00071276 2023-10-20T03:36:24.787 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.788 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.788 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.788 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889488+0000", 2023-10-20T03:36:24.788 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5309000000000002e-05 2023-10-20T03:36:24.788 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.789 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.789 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.789 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889511+0000", 2023-10-20T03:36:24.789 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3572999999999999e-05 2023-10-20T03:36:24.789 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.789 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.790 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.790 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889746+0000", 2023-10-20T03:36:24.790 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00023487099999999999 2023-10-20T03:36:24.790 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.791 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.791 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.791 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.889773+0000", 2023-10-20T03:36:24.791 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7330000000000001e-05 2023-10-20T03:36:24.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.792 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.792 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.792 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.915601+0000", 2023-10-20T03:36:24.792 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.02582746 2023-10-20T03:36:24.792 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.793 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.793 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.793 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:41.915729+0000", 2023-10-20T03:36:24.793 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012847100000000001 2023-10-20T03:36:24.793 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.794 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.794 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.794 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.794 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.794 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:589 3.es0 3:7b3c804c:::smithi17039300-40:head [append 0~720896] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:24.795 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.001880+0000", 2023-10-20T03:36:24.795 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.696231152999999, 2023-10-20T03:36:24.795 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.047597967999999997, 2023-10-20T03:36:24.795 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.795 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.796 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.796 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.796 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.796 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 589 2023-10-20T03:36:24.797 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.797 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.797 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.797 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.797 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.001880+0000", 2023-10-20T03:36:24.798 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.798 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.798 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.798 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.798 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.001880+0000", 2023-10-20T03:36:24.799 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.799 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.799 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.799 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.799 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.001882+0000", 2023-10-20T03:36:24.800 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2139999999999999e-06 2023-10-20T03:36:24.800 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.800 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.800 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.801 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.002268+0000", 2023-10-20T03:36:24.801 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00038638700000000003 2023-10-20T03:36:24.801 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.801 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.801 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.802 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.002297+0000", 2023-10-20T03:36:24.802 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8697e-05 2023-10-20T03:36:24.802 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.802 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.802 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.803 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.002348+0000", 2023-10-20T03:36:24.803 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1187999999999998e-05 2023-10-20T03:36:24.803 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.803 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.803 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.804 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.002561+0000", 2023-10-20T03:36:24.804 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000213427 2023-10-20T03:36:24.804 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.804 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.804 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.804 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.002733+0000", 2023-10-20T03:36:24.805 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000171256 2023-10-20T03:36:24.805 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.805 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.805 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.805 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.004158+0000", 2023-10-20T03:36:24.806 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0014255330000000001 2023-10-20T03:36:24.806 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.806 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.806 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.806 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.039236+0000", 2023-10-20T03:36:24.807 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.035078249999999998 2023-10-20T03:36:24.807 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.807 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.807 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.807 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.049317+0000", 2023-10-20T03:36:24.808 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010080458 2023-10-20T03:36:24.808 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.808 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.808 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.808 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.049478+0000", 2023-10-20T03:36:24.809 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016055799999999999 2023-10-20T03:36:24.809 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.809 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.809 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.809 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.810 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.810 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:590 3.es0 3:7b3c804c:::smithi17039300-40:head [append 0~802816] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:24.810 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.009044+0000", 2023-10-20T03:36:24.810 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.689066423, 2023-10-20T03:36:24.811 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.041318410999999999, 2023-10-20T03:36:24.811 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.811 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.811 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.811 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.812 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.812 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 590 2023-10-20T03:36:24.812 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.812 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.812 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.813 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.813 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.009044+0000", 2023-10-20T03:36:24.813 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.813 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.813 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.814 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.814 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.009044+0000", 2023-10-20T03:36:24.814 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.814 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.814 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.815 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.815 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.009047+0000", 2023-10-20T03:36:24.815 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6879999999999999e-06 2023-10-20T03:36:24.815 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.815 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.816 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.816 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.009793+0000", 2023-10-20T03:36:24.816 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00074570099999999998 2023-10-20T03:36:24.816 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.816 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.817 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.817 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.009828+0000", 2023-10-20T03:36:24.817 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5045000000000001e-05 2023-10-20T03:36:24.817 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.009851+0000", 2023-10-20T03:36:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3206000000000001e-05 2023-10-20T03:36:24.818 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.011468+0000", 2023-10-20T03:36:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001617402 2023-10-20T03:36:24.819 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.011751+0000", 2023-10-20T03:36:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000283107 2023-10-20T03:36:24.820 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.014038+0000", 2023-10-20T03:36:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002286695 2023-10-20T03:36:24.821 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.039572+0000", 2023-10-20T03:36:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025533541999999999 2023-10-20T03:36:24.822 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.049801+0000", 2023-10-20T03:36:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010229548999999999 2023-10-20T03:36:24.823 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.824 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.824 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.824 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.050363+0000", 2023-10-20T03:36:24.824 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00056147599999999999 2023-10-20T03:36:24.824 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.825 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:591 3.es0 3:7b3c804c:::smithi17039300-40:head [append 0~360448] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.012685+0000", 2023-10-20T03:36:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.685426144000001, 2023-10-20T03:36:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.038593720999999998, 2023-10-20T03:36:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.826 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 591 2023-10-20T03:36:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.827 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012685+0000", 2023-10-20T03:36:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.828 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012685+0000", 2023-10-20T03:36:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.829 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012686+0000", 2023-10-20T03:36:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.429e-06 2023-10-20T03:36:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.830 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.831 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.831 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012926+0000", 2023-10-20T03:36:24.831 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024043099999999999 2023-10-20T03:36:24.831 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.831 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.832 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.832 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012940+0000", 2023-10-20T03:36:24.832 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3447e-05 2023-10-20T03:36:24.832 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.832 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.833 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.833 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012947+0000", 2023-10-20T03:36:24.833 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4590000000000001e-06 2023-10-20T03:36:24.833 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.834 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.834 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.834 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.034646+0000", 2023-10-20T03:36:24.834 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.021698205000000002 2023-10-20T03:36:24.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.835 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.835 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.835 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.034733+0000", 2023-10-20T03:36:24.835 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7528000000000006e-05 2023-10-20T03:36:24.835 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.836 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.836 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.836 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.035416+0000", 2023-10-20T03:36:24.836 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00068264400000000005 2023-10-20T03:36:24.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.837 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.837 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.837 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.042809+0000", 2023-10-20T03:36:24.837 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0073934220000000002 2023-10-20T03:36:24.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.838 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.838 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.838 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.050705+0000", 2023-10-20T03:36:24.838 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0078962830000000005 2023-10-20T03:36:24.838 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.839 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.839 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.839 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.051278+0000", 2023-10-20T03:36:24.839 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000572873 2023-10-20T03:36:24.839 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.840 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.840 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.840 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.840 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.840 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:592 3.es0 3:7b3c804c:::smithi17039300-40:head [setxattr _header (57) in=7b] snapc 23=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:24.841 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.012969+0000", 2023-10-20T03:36:24.841 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.685141326, 2023-10-20T03:36:24.841 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039254223999999997, 2023-10-20T03:36:24.841 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.841 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.842 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.842 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.842 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.842 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 592 2023-10-20T03:36:24.843 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.843 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.843 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.843 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.843 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012969+0000", 2023-10-20T03:36:24.844 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.844 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.844 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.844 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.844 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012969+0000", 2023-10-20T03:36:24.845 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.845 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.845 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.845 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.845 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012970+0000", 2023-10-20T03:36:24.846 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4799999999999997e-07 2023-10-20T03:36:24.846 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.846 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.846 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.847 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012975+0000", 2023-10-20T03:36:24.847 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6160000000000004e-06 2023-10-20T03:36:24.847 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.847 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.847 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.848 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012984+0000", 2023-10-20T03:36:24.848 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0850000000000003e-06 2023-10-20T03:36:24.848 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.848 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.848 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.848 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.012992+0000", 2023-10-20T03:36:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6270000000000006e-06 2023-10-20T03:36:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.849 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.039656+0000", 2023-10-20T03:36:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.02666342 2023-10-20T03:36:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.850 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.039948+0000", 2023-10-20T03:36:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000292458 2023-10-20T03:36:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.041465+0000", 2023-10-20T03:36:24.851 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00151659 2023-10-20T03:36:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.043158+0000", 2023-10-20T03:36:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0016926210000000001 2023-10-20T03:36:24.852 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.051619+0000", 2023-10-20T03:36:24.853 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0084616410000000006 2023-10-20T03:36:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.854 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.052224+0000", 2023-10-20T03:36:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00060441800000000001 2023-10-20T03:36:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.855 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:593 3.es0 3:7b3c804c:::smithi17039300-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2023-10-20T03:36:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:35:53.013003+0000", 2023-10-20T03:36:24.856 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 31.68510805, 2023-10-20T03:36:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.041417311999999998, 2023-10-20T03:36:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.857 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 593 2023-10-20T03:36:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.858 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013003+0000", 2023-10-20T03:36:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.859 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013003+0000", 2023-10-20T03:36:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.860 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013003+0000", 2023-10-20T03:36:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8299999999999997e-07 2023-10-20T03:36:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.861 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013006+0000", 2023-10-20T03:36:24.862 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6699999999999998e-06 2023-10-20T03:36:24.862 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.862 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.862 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.862 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013015+0000", 2023-10-20T03:36:24.863 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3479999999999993e-06 2023-10-20T03:36:24.863 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.863 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.863 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.863 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.013021+0000", 2023-10-20T03:36:24.864 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7880000000000003e-06 2023-10-20T03:36:24.864 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.864 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.864 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.864 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.042224+0000", 2023-10-20T03:36:24.865 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029202676 2023-10-20T03:36:24.865 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.865 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.865 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:24.865 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.042562+0000", 2023-10-20T03:36:24.866 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000338029 2023-10-20T03:36:24.866 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.866 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.866 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.866 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.053183+0000", 2023-10-20T03:36:24.867 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010621619000000001 2023-10-20T03:36:24.867 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.867 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.867 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.867 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.053311+0000", 2023-10-20T03:36:24.867 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012811800000000001 2023-10-20T03:36:24.868 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.868 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.868 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.868 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:35:53.054420+0000", 2023-10-20T03:36:24.868 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0011084809999999999 2023-10-20T03:36:24.868 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.869 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.869 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.869 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.869 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.869 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:713 3.es0 3:7b3c804c:::smithi17039300-40:head [append 0~589824] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2023-10-20T03:36:24.870 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:36:08.118934+0000", 2023-10-20T03:36:24.870 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 16.579176308000001, 2023-10-20T03:36:24.870 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.046991040999999997, 2023-10-20T03:36:24.870 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.870 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.871 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.871 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.871 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.871 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 713 2023-10-20T03:36:24.871 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.872 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.872 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.872 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.872 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.118934+0000", 2023-10-20T03:36:24.872 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.873 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.873 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.873 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.873 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.118934+0000", 2023-10-20T03:36:24.873 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.874 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.874 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.874 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.874 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.118937+0000", 2023-10-20T03:36:24.875 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4499999999999998e-06 2023-10-20T03:36:24.875 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.875 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.875 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.875 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.119699+0000", 2023-10-20T03:36:24.876 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00076246599999999995 2023-10-20T03:36:24.876 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.876 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.876 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.876 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.119732+0000", 2023-10-20T03:36:24.877 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3068000000000003e-05 2023-10-20T03:36:24.877 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.877 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.877 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.877 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.119747+0000", 2023-10-20T03:36:24.878 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4897e-05 2023-10-20T03:36:24.878 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.878 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.878 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.878 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.119872+0000", 2023-10-20T03:36:24.878 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012461500000000001 2023-10-20T03:36:24.879 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.879 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.879 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.879 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.120059+0000", 2023-10-20T03:36:24.879 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018735799999999999 2023-10-20T03:36:24.880 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.880 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.880 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.880 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.121905+0000", 2023-10-20T03:36:24.880 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001845274 2023-10-20T03:36:24.881 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.881 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.881 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.881 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.146206+0000", 2023-10-20T03:36:24.881 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.024301335 2023-10-20T03:36:24.881 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.882 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.882 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.882 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.165362+0000", 2023-10-20T03:36:24.882 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019156177999999999 2023-10-20T03:36:24.882 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.883 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.883 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.883 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.165925+0000", 2023-10-20T03:36:24.883 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00056340000000000003 2023-10-20T03:36:24.883 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.884 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.884 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.884 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.884 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.884 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:714 3.es0 3:7b3c804c:::smithi17039300-40:head [append 0~753664] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2023-10-20T03:36:24.885 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:36:08.129996+0000", 2023-10-20T03:36:24.885 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 16.568114484999999, 2023-10-20T03:36:24.885 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039326891000000003, 2023-10-20T03:36:24.885 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.885 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.886 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.886 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 714 2023-10-20T03:36:24.886 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.887 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.887 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.887 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.887 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.129996+0000", 2023-10-20T03:36:24.887 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.888 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.888 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.888 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.888 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.129996+0000", 2023-10-20T03:36:24.888 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.889 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.889 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.889 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.889 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.129999+0000", 2023-10-20T03:36:24.889 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9129999999999999e-06 2023-10-20T03:36:24.889 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.130972+0000", 2023-10-20T03:36:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00097279400000000002 2023-10-20T03:36:24.890 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.131006+0000", 2023-10-20T03:36:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3608000000000001e-05 2023-10-20T03:36:24.891 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.131020+0000", 2023-10-20T03:36:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4846e-05 2023-10-20T03:36:24.892 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.132387+0000", 2023-10-20T03:36:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001366319 2023-10-20T03:36:24.893 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.132697+0000", 2023-10-20T03:36:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00031011100000000001 2023-10-20T03:36:24.894 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.134733+0000", 2023-10-20T03:36:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020364879999999999 2023-10-20T03:36:24.895 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.164780+0000", 2023-10-20T03:36:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030046691 2023-10-20T03:36:24.896 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.168889+0000", 2023-10-20T03:36:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0041086869999999998 2023-10-20T03:36:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.897 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.169323+0000", 2023-10-20T03:36:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00043443399999999998 2023-10-20T03:36:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.898 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:715 3.es0 3:7b3c804c:::smithi17039300-40:head [append 0~475136] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2023-10-20T03:36:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:36:08.136135+0000", 2023-10-20T03:36:24.899 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 16.561975692000001, 2023-10-20T03:36:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.033855028000000002, 2023-10-20T03:36:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.900 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 715 2023-10-20T03:36:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.901 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.136135+0000", 2023-10-20T03:36:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.902 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.136135+0000", 2023-10-20T03:36:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.903 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.904 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.904 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.904 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.136138+0000", 2023-10-20T03:36:24.904 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.01e-06 2023-10-20T03:36:24.904 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.905 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.905 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.905 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.136805+0000", 2023-10-20T03:36:24.905 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00066697299999999996 2023-10-20T03:36:24.905 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.906 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.906 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.906 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.136841+0000", 2023-10-20T03:36:24.906 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5840999999999998e-05 2023-10-20T03:36:24.907 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.907 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.907 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.907 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.136857+0000", 2023-10-20T03:36:24.907 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5854000000000001e-05 2023-10-20T03:36:24.908 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.908 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.908 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.908 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.146744+0000", 2023-10-20T03:36:24.908 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0098877930000000006 2023-10-20T03:36:24.909 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.909 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.909 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.909 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.147068+0000", 2023-10-20T03:36:24.909 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00032364699999999999 2023-10-20T03:36:24.909 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.910 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.910 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.910 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.148710+0000", 2023-10-20T03:36:24.910 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0016417269999999999 2023-10-20T03:36:24.910 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.911 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.911 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.911 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.164782+0000", 2023-10-20T03:36:24.911 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016072258999999998 2023-10-20T03:36:24.911 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.911 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.912 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.912 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.169729+0000", 2023-10-20T03:36:24.912 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0049473369999999996 2023-10-20T03:36:24.912 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.912 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.913 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.913 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.169990+0000", 2023-10-20T03:36:24.913 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000260587 2023-10-20T03:36:24.913 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.913 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.914 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.914 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.914 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.914 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:716 3.es0 3:7b3c804c:::smithi17039300-40:head [append 0~327680] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2023-10-20T03:36:24.914 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:36:08.139405+0000", 2023-10-20T03:36:24.915 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 16.558705232000001, 2023-10-20T03:36:24.915 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.032093397000000003, 2023-10-20T03:36:24.915 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.915 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.915 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.916 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.916 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.916 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 716 2023-10-20T03:36:24.916 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.916 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.917 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.917 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.917 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.139405+0000", 2023-10-20T03:36:24.917 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.917 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.918 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.918 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.918 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.139405+0000", 2023-10-20T03:36:24.918 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.919 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.919 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.919 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.919 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.139408+0000", 2023-10-20T03:36:24.919 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7470000000000001e-06 2023-10-20T03:36:24.920 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.920 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.920 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.920 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.139900+0000", 2023-10-20T03:36:24.920 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000492287 2023-10-20T03:36:24.920 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.921 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.921 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.921 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.139934+0000", 2023-10-20T03:36:24.921 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3578000000000003e-05 2023-10-20T03:36:24.921 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.922 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.922 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.922 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.139949+0000", 2023-10-20T03:36:24.922 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5082999999999999e-05 2023-10-20T03:36:24.922 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.923 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.923 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.923 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.156167+0000", 2023-10-20T03:36:24.923 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016218257 2023-10-20T03:36:24.923 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.924 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.924 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.924 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.156525+0000", 2023-10-20T03:36:24.924 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000357785 2023-10-20T03:36:24.924 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.925 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.925 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.925 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.158176+0000", 2023-10-20T03:36:24.925 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0016505929999999999 2023-10-20T03:36:24.925 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.926 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.926 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.926 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.166269+0000", 2023-10-20T03:36:24.926 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0080933159999999997 2023-10-20T03:36:24.926 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.927 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.927 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.927 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.171094+0000", 2023-10-20T03:36:24.927 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0048249440000000003 2023-10-20T03:36:24.927 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.928 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.928 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.928 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.171499+0000", 2023-10-20T03:36:24.928 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00040480699999999998 2023-10-20T03:36:24.928 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.929 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.929 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.929 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.929 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.929 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:717 3.es0 3:7b3c804c:::smithi17039300-40:head [setxattr _header (57) in=7b] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2023-10-20T03:36:24.930 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:36:08.139977+0000", 2023-10-20T03:36:24.930 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 16.558133264999999, 2023-10-20T03:36:24.930 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.032412565999999997, 2023-10-20T03:36:24.930 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.930 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.931 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.931 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.931 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.931 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 717 2023-10-20T03:36:24.931 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.932 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.932 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.932 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.932 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.139977+0000", 2023-10-20T03:36:24.932 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.933 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.933 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.933 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.933 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.139977+0000", 2023-10-20T03:36:24.933 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.934 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.934 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.934 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.934 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.139986+0000", 2023-10-20T03:36:24.935 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3459999999999994e-06 2023-10-20T03:36:24.935 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.935 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.935 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.935 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.139991+0000", 2023-10-20T03:36:24.935 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6030000000000004e-06 2023-10-20T03:36:24.936 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.936 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.936 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.936 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.140003+0000", 2023-10-20T03:36:24.936 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.183e-05 2023-10-20T03:36:24.937 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.937 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.937 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.937 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.140012+0000", 2023-10-20T03:36:24.937 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4489999999999999e-06 2023-10-20T03:36:24.938 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.938 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.938 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.938 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.166658+0000", 2023-10-20T03:36:24.938 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026645973999999999 2023-10-20T03:36:24.939 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.939 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.939 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.939 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.166825+0000", 2023-10-20T03:36:24.939 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000166973 2023-10-20T03:36:24.939 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.940 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.940 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.940 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.167557+0000", 2023-10-20T03:36:24.940 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00073213 2023-10-20T03:36:24.940 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.941 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.941 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.941 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.169555+0000", 2023-10-20T03:36:24.941 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001998054 2023-10-20T03:36:24.941 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.942 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.942 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.942 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.172060+0000", 2023-10-20T03:36:24.942 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0025056459999999998 2023-10-20T03:36:24.942 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.942 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.943 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.943 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.172390+0000", 2023-10-20T03:36:24.943 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00032956100000000002 2023-10-20T03:36:24.943 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.943 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.944 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.944 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.944 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.944 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:718 3.es0 3:7b3c804c:::smithi17039300-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2023-10-20T03:36:24.944 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:36:08.140025+0000", 2023-10-20T03:36:24.944 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 16.558086057000001, 2023-10-20T03:36:24.945 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.035763433999999997, 2023-10-20T03:36:24.945 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.945 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2023-10-20T03:36:24.945 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.945 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.946 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.946 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 718 2023-10-20T03:36:24.946 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.946 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.946 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.946 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.947 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.140025+0000", 2023-10-20T03:36:24.947 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.947 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.947 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.947 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.948 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.140025+0000", 2023-10-20T03:36:24.948 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.948 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.948 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.948 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.949 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.140025+0000", 2023-10-20T03:36:24.949 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6300000000000004e-07 2023-10-20T03:36:24.949 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.949 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.949 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.950 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.140029+0000", 2023-10-20T03:36:24.950 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2169999999999999e-06 2023-10-20T03:36:24.950 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.950 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.950 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.950 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.140039+0000", 2023-10-20T03:36:24.951 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0563e-05 2023-10-20T03:36:24.951 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.951 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.951 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.951 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.140047+0000", 2023-10-20T03:36:24.952 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6459999999999993e-06 2023-10-20T03:36:24.952 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.952 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.952 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.953 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.168196+0000", 2023-10-20T03:36:24.953 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.028149528 2023-10-20T03:36:24.953 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.953 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.953 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:24.954 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.168501+0000", 2023-10-20T03:36:24.954 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00030487299999999999 2023-10-20T03:36:24.954 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.954 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.954 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.955 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.173026+0000", 2023-10-20T03:36:24.955 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0045244830000000002 2023-10-20T03:36:24.955 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.955 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.955 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.956 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.173348+0000", 2023-10-20T03:36:24.956 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00032241399999999998 2023-10-20T03:36:24.956 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.956 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.956 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.957 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:08.175788+0000", 2023-10-20T03:36:24.957 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0024398470000000002 2023-10-20T03:36:24.957 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.957 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.957 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.958 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.958 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.958 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:842 3.es0 3:7dffa048:::smithi17039300-5:head [rollback 38] snapc 3b=[38] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-10-20T03:36:24.958 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:36:19.190913+0000", 2023-10-20T03:36:24.958 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 5.5071979239999997, 2023-10-20T03:36:24.959 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.02586772, 2023-10-20T03:36:24.959 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.959 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.959 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.959 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.960 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.960 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 842 2023-10-20T03:36:24.960 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.960 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.960 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.961 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.961 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.190913+0000", 2023-10-20T03:36:24.961 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.961 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.961 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.961 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.962 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.190913+0000", 2023-10-20T03:36:24.962 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.962 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.962 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.963 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.963 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.190914+0000", 2023-10-20T03:36:24.963 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.23e-07 2023-10-20T03:36:24.963 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.963 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.964 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.964 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.190918+0000", 2023-10-20T03:36:24.964 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4909999999999999e-06 2023-10-20T03:36:24.964 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.964 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.965 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.965 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.190930+0000", 2023-10-20T03:36:24.965 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.207e-05 2023-10-20T03:36:24.965 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.966 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.966 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.966 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.190946+0000", 2023-10-20T03:36:24.966 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.554e-05 2023-10-20T03:36:24.966 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.967 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.967 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.967 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.193730+0000", 2023-10-20T03:36:24.967 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0027847340000000001 2023-10-20T03:36:24.967 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.968 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.968 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.968 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.193852+0000", 2023-10-20T03:36:24.968 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000121698 2023-10-20T03:36:24.968 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.969 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.969 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.969 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.194525+0000", 2023-10-20T03:36:24.969 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00067287399999999995 2023-10-20T03:36:24.969 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.970 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.970 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.970 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.199670+0000", 2023-10-20T03:36:24.970 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0051446900000000004 2023-10-20T03:36:24.970 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.970 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.971 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.971 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.216690+0000", 2023-10-20T03:36:24.971 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.017019956999999999 2023-10-20T03:36:24.971 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.972 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.972 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.972 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.216780+0000", 2023-10-20T03:36:24.972 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0842999999999997e-05 2023-10-20T03:36:24.972 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.973 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.973 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.973 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.973 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.973 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.4570.0:843 3.es0 3:7dffa048:::smithi17039300-5:head [append 0~0] snapc 3b=[38] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-10-20T03:36:24.973 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2023-10-20T03:36:19.190960+0000", 2023-10-20T03:36:24.974 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 5.5071509379999997, 2023-10-20T03:36:24.974 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026083288999999999, 2023-10-20T03:36:24.974 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2023-10-20T03:36:24.974 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:24.975 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2023-10-20T03:36:24.975 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.4570", 2023-10-20T03:36:24.975 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:24.975 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 843 2023-10-20T03:36:24.975 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.975 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2023-10-20T03:36:24.976 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.976 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2023-10-20T03:36:24.976 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.190960+0000", 2023-10-20T03:36:24.976 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.976 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.977 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.977 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2023-10-20T03:36:24.977 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.190960+0000", 2023-10-20T03:36:24.977 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2023-10-20T03:36:24.977 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.978 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.978 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2023-10-20T03:36:24.978 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.190960+0000", 2023-10-20T03:36:24.978 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3700000000000001e-07 2023-10-20T03:36:24.978 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.979 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.979 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2023-10-20T03:36:24.979 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.190964+0000", 2023-10-20T03:36:24.979 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5999999999999998e-06 2023-10-20T03:36:24.979 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.980 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.980 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2023-10-20T03:36:24.980 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.190970+0000", 2023-10-20T03:36:24.980 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4180000000000002e-06 2023-10-20T03:36:24.980 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.981 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.981 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2023-10-20T03:36:24.981 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.190975+0000", 2023-10-20T03:36:24.981 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5959999999999997e-06 2023-10-20T03:36:24.981 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.982 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.982 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2023-10-20T03:36:24.982 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.196184+0000", 2023-10-20T03:36:24.982 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0052092370000000002 2023-10-20T03:36:24.982 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.983 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.983 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2023-10-20T03:36:24.983 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.196532+0000", 2023-10-20T03:36:24.983 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00034764300000000002 2023-10-20T03:36:24.983 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.984 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.984 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_started", 2023-10-20T03:36:24.984 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.198108+0000", 2023-10-20T03:36:24.984 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001576643 2023-10-20T03:36:24.984 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.984 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.985 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_committed", 2023-10-20T03:36:24.985 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.200615+0000", 2023-10-20T03:36:24.985 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0025065180000000001 2023-10-20T03:36:24.985 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.985 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.986 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2023-10-20T03:36:24.986 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.216934+0000", 2023-10-20T03:36:24.986 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016319074999999999 2023-10-20T03:36:24.986 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2023-10-20T03:36:24.986 INFO:teuthology.orchestra.run.smithi170.stdout: { 2023-10-20T03:36:24.987 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2023-10-20T03:36:24.987 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2023-10-20T03:36:19.217043+0000", 2023-10-20T03:36:24.987 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000109122 2023-10-20T03:36:24.987 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.987 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.987 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.988 INFO:teuthology.orchestra.run.smithi170.stdout: } 2023-10-20T03:36:24.988 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2023-10-20T03:36:24.988 INFO:teuthology.orchestra.run.smithi170.stdout:} 2023-10-20T03:36:24.989 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:36:24.990 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:36:24.991 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:36:24.991 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:36:24.991 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-10-20T03:36:24.992 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2023-10-20T03:36:24.993 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-10-20T03:36:24.993 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:36:24.993 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:36:24.994 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:36:24.994 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:36:24.995 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.9 2023-10-20T03:36:24.995 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:36:24.996 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:36:24.984+0000 7f47dadc5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:36:24.996 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:36:25.037 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:36:25.037 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:36:25.063 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:36:25.063 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:36:25.091 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:25.088+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:25.096 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:36:25.096 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:36:25.125 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:36:25.126 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-10-20T03:36:25.152 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:36:25.152 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:36:25.156 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:36:25.157 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:36:25.167 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:25.167 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:25.181 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-10-20T03:36:25.192 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:36:25.191+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:36:25.263 INFO:tasks.rados.rados.0.smithi170.stdout:389: done (10 left) 2023-10-20T03:36:25.263 INFO:tasks.rados.rados.0.smithi170.stdout:390: copy_from oid 2 from oid 3 current snap is 34 2023-10-20T03:36:25.263 INFO:tasks.rados.rados.0.smithi170.stdout:387: finishing write tid 1 to smithi17039300-6 2023-10-20T03:36:25.263 INFO:tasks.rados.rados.0.smithi170.stdout:378: expect (ObjNum 91 snap 11 seq_num 91) 2023-10-20T03:36:25.293 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:25.291+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:25.316 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:25.316 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:25.330 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-10-20T03:36:25.363 INFO:tasks.rados.rados.0.smithi170.stdout:383: finishing rollback tid 0 to smithi17039300-35 2023-10-20T03:36:25.363 INFO:tasks.rados.rados.0.smithi170.stdout:383: finishing rollback tid 1 to smithi17039300-35 2023-10-20T03:36:25.363 INFO:tasks.rados.rados.0.smithi170.stdout:384: finishing write tid 1 to smithi17039300-12 2023-10-20T03:36:25.363 INFO:tasks.rados.rados.0.smithi170.stdout:380: finishing rollback tid 1 to smithi17039300-28 2023-10-20T03:36:25.363 INFO:tasks.rados.rados.0.smithi170.stdout:384: finishing write tid 2 to smithi17039300-12 2023-10-20T03:36:25.364 INFO:tasks.rados.rados.0.smithi170.stdout:383: finishing rollback tid 2 to smithi17039300-35 2023-10-20T03:36:25.364 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 35 v 72 (ObjNum 149 snap 33 seq_num 149) dirty exists 2023-10-20T03:36:25.364 INFO:tasks.rados.rados.0.smithi170.stdout:387: finishing write tid 2 to smithi17039300-6 2023-10-20T03:36:25.364 INFO:tasks.rados.rados.0.smithi170.stdout:378: done (10 left) 2023-10-20T03:36:25.364 INFO:tasks.rados.rados.0.smithi170.stdout:383: done (9 left) 2023-10-20T03:36:25.365 INFO:tasks.rados.rados.0.smithi170.stdout:391: read oid 19 snap 33 2023-10-20T03:36:25.365 INFO:tasks.rados.rados.0.smithi170.stdout:391: expect (ObjNum 120 snap 25 seq_num 120) 2023-10-20T03:36:25.365 INFO:tasks.rados.rados.0.smithi170.stdout:392: snap_create 2023-10-20T03:36:25.365 INFO:tasks.rados.rados.0.smithi170.stdout:386: finishing write tid 1 to smithi17039300-43 2023-10-20T03:36:25.365 INFO:tasks.rados.rados.0.smithi170.stdout:381: expect (ObjNum 78 snap 7 seq_num 78) 2023-10-20T03:36:25.394 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:25.388+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:25.419 INFO:tasks.rados.rados.0.smithi170.stdout:380: finishing rollback tid 2 to smithi17039300-28 2023-10-20T03:36:25.419 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 28 v 91 (ObjNum 94 snap 13 seq_num 94) dirty exists 2023-10-20T03:36:25.420 INFO:tasks.rados.rados.0.smithi170.stdout:385: expect (ObjNum 105 snap 16 seq_num 105) 2023-10-20T03:36:25.448 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:25.448 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:25.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-10-20T03:36:25.470 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 22 v 68 (ObjNum 152 snap 33 seq_num 152) dirty exists 2023-10-20T03:36:25.471 INFO:tasks.rados.rados.0.smithi170.stdout:384: finishing write tid 3 to smithi17039300-12 2023-10-20T03:36:25.471 INFO:tasks.rados.rados.0.smithi170.stdout:384: oid 12 updating version 0 to 15 2023-10-20T03:36:25.471 INFO:tasks.rados.rados.0.smithi170.stdout:384: oid 12 version 15 is already newer than 14 2023-10-20T03:36:25.471 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 12 v 15 (ObjNum 153 snap 34 seq_num 153) dirty exists 2023-10-20T03:36:25.472 INFO:tasks.rados.rados.0.smithi170.stdout:384: left oid 12 (ObjNum 153 snap 34 seq_num 153) 2023-10-20T03:36:25.472 INFO:tasks.rados.rados.0.smithi170.stdout:386: finishing write tid 2 to smithi17039300-43 2023-10-20T03:36:25.472 INFO:tasks.rados.rados.0.smithi170.stdout:387: finishing write tid 3 to smithi17039300-6 2023-10-20T03:36:25.472 INFO:tasks.rados.rados.0.smithi170.stdout:387: oid 6 updating version 0 to 7 2023-10-20T03:36:25.472 INFO:tasks.rados.rados.0.smithi170.stdout:387: oid 6 updating version 7 to 8 2023-10-20T03:36:25.472 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 6 v 8 (ObjNum 155 snap 34 seq_num 155) dirty exists 2023-10-20T03:36:25.473 INFO:tasks.rados.rados.0.smithi170.stdout:387: left oid 6 (ObjNum 155 snap 34 seq_num 155) 2023-10-20T03:36:25.473 INFO:tasks.rados.rados.0.smithi170.stdout:386: finishing write tid 3 to smithi17039300-43 2023-10-20T03:36:25.473 INFO:tasks.rados.rados.0.smithi170.stdout:386: oid 43 updating version 0 to 94 2023-10-20T03:36:25.473 INFO:tasks.rados.rados.0.smithi170.stdout:386: oid 43 version 94 is already newer than 93 2023-10-20T03:36:25.473 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 43 v 94 (ObjNum 154 snap 34 seq_num 154) dirty exists 2023-10-20T03:36:25.474 INFO:tasks.rados.rados.0.smithi170.stdout:386: left oid 43 (ObjNum 154 snap 34 seq_num 154) 2023-10-20T03:36:25.474 INFO:tasks.rados.rados.0.smithi170.stdout:377: finishing copy_from racing read to smithi17039300-4 2023-10-20T03:36:25.474 INFO:tasks.rados.rados.0.smithi170.stdout:377: finishing copy_from to smithi17039300-4 2023-10-20T03:36:25.474 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 4 v 56 (ObjNum 150 snap 33 seq_num 150) dirty exists 2023-10-20T03:36:25.474 INFO:tasks.rados.rados.0.smithi170.stdout:390: finishing copy_from racing read to smithi17039300-2 2023-10-20T03:36:25.475 INFO:tasks.rados.rados.0.smithi170.stdout:390: finishing copy_from to smithi17039300-2 2023-10-20T03:36:25.475 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 2 v 86 (ObjNum 2 snap 0 seq_num 2) dirty exists 2023-10-20T03:36:25.475 INFO:tasks.rados.rados.0.smithi170.stdout:391: expect (ObjNum 120 snap 25 seq_num 120) 2023-10-20T03:36:25.494 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:25.492+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:25.596 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:25.592+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:25.596 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:25.597 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:25.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-10-20T03:36:25.697 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:25.696+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:25.733 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:25.733 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:25.747 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-10-20T03:36:25.797 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:25.796+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:25.875 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:25.875 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:25.889 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-10-20T03:36:25.898 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:25.895+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:25.999 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:36:25.995+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:36:26.023 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:26.023 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:26.038 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-10-20T03:36:26.100 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:26.096+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:26.155 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:26.156 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:26.170 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-10-20T03:36:26.201 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:26.196+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:26.275 INFO:tasks.rados.rados.0.smithi170.stdout:377: done (10 left) 2023-10-20T03:36:26.275 INFO:tasks.rados.rados.0.smithi170.stdout:380: done (9 left) 2023-10-20T03:36:26.275 INFO:tasks.rados.rados.0.smithi170.stdout:381: done (8 left) 2023-10-20T03:36:26.275 INFO:tasks.rados.rados.0.smithi170.stdout:384: done (7 left) 2023-10-20T03:36:26.276 INFO:tasks.rados.rados.0.smithi170.stdout:385: done (6 left) 2023-10-20T03:36:26.276 INFO:tasks.rados.rados.0.smithi170.stdout:386: done (5 left) 2023-10-20T03:36:26.276 INFO:tasks.rados.rados.0.smithi170.stdout:387: done (4 left) 2023-10-20T03:36:26.276 INFO:tasks.rados.rados.0.smithi170.stdout:388: done (3 left) 2023-10-20T03:36:26.276 INFO:tasks.rados.rados.0.smithi170.stdout:390: done (2 left) 2023-10-20T03:36:26.276 INFO:tasks.rados.rados.0.smithi170.stdout:391: done (1 left) 2023-10-20T03:36:26.277 INFO:tasks.rados.rados.0.smithi170.stdout:392: done (0 left) 2023-10-20T03:36:26.277 INFO:tasks.rados.rados.0.smithi170.stdout:393: snap_remove snap 33 2023-10-20T03:36:26.277 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-10-20T03:36:26.292 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:26.292 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:26.301 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:26.296+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:26.307 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-10-20T03:36:26.402 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:26.400+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:26.479 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:26.479 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:26.494 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-10-20T03:36:26.504 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:26.500+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:26.604 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:26.603+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:26.654 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:26.654 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:26.667 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-10-20T03:36:26.705 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:26.703+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:26.790 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:26.790 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:26.804 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-10-20T03:36:26.806 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:26.800+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:26.907 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:26.904+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:26.931 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:26.931 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:26.945 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-10-20T03:36:27.008 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:36:27.007+0000 7f7212305640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:36:27.063 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:27.063 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:27.077 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-10-20T03:36:27.110 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:27.104+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:27.157 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-10-20T03:36:27.202 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:27.203 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:27.210 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:36:27.208+0000 7f8061d5f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:36:27.217 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-10-20T03:36:27.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~8s 2023-10-20T03:36:27.241 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15] out_osds: [12] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 10, 14, 3, 7, 11, 15, 6] 2023-10-20T03:36:27.242 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-10-20T03:36:27.242 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2023-10-20T03:36:27.242 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-10-20T03:36:27.242 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-10-20T03:36:27.243 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok config set filestore_inject_stall 3 2023-10-20T03:36:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is set 2023-10-20T03:36:27.278 INFO:tasks.rados.rados.0.smithi170.stdout:393: done (0 left) 2023-10-20T03:36:27.278 INFO:tasks.rados.rados.0.smithi170.stdout:394: snap_remove snap 34 2023-10-20T03:36:27.311 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:36:27.312 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.312 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:27.308+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:27.333 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:36:27.333 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.368 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:36:27.368 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.386 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:27.387 INFO:teuthology.orchestra.run.smithi033.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-10-20T03:36:27.387 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:27.388 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:36:27.388 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.396 INFO:teuthology.orchestra.run.smithi033.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:27.397 INFO:teuthology.orchestra.run.smithi033.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:27.406 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:36:27.406 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.413 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-10-20T03:36:27.413 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:27.408+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:27.426 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:36:27.428 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.447 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2023-10-20T03:36:27.447 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.467 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:36:27.468 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.483 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:36:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.484 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.9 2023-10-20T03:36:27.503 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:36:27.504 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.513 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:36:27.508+0000 7f1813d72640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:36:27.523 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:36:27.524 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.543 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:36:27.544 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.544 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:27.544 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [], 2023-10-20T03:36:27.544 INFO:teuthology.orchestra.run.smithi033.stdout: "complaint_time": 30, 2023-10-20T03:36:27.545 INFO:teuthology.orchestra.run.smithi033.stdout: "num_blocked_ops": 0 2023-10-20T03:36:27.545 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:27.558 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-10-20T03:36:27.564 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:36:27.564 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.586 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:36:27.586 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.607 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:36:27.608 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-10-20T03:36:27.611 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:36:27.612 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:36:27.613 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:27.608+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:27.690 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-10-20T03:36:27.690 INFO:teuthology.orchestra.run.smithi033.stdout: "size": 20, 2023-10-20T03:36:27.691 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 600, 2023-10-20T03:36:27.691 INFO:teuthology.orchestra.run.smithi033.stdout: "ops": [ 2023-10-20T03:36:27.691 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.691 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=82, reqid=client.4570.0:311, at_version=43'24, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:36:27.691 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:36:27.692 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 62.330365694999998, 2023-10-20T03:36:27.692 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010383547999999999, 2023-10-20T03:36:27.692 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.692 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:27.692 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.693 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.693 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.693 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:36:27.693 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.693 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.693 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.694 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.694 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356839+0000", 2023-10-20T03:36:27.694 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.694 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.694 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.695 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.356840+0000", 2023-10-20T03:36:27.695 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.1999999999999999e-07 2023-10-20T03:36:27.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.695 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.695 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.696 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357506+0000", 2023-10-20T03:36:27.696 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00066620500000000001 2023-10-20T03:36:27.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.696 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.696 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.697 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357522+0000", 2023-10-20T03:36:27.697 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5964000000000002e-05 2023-10-20T03:36:27.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.697 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.697 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.698 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357535+0000", 2023-10-20T03:36:27.698 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3046e-05 2023-10-20T03:36:27.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.698 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.698 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.699 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357625+0000", 2023-10-20T03:36:27.699 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.9691999999999994e-05 2023-10-20T03:36:27.699 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.699 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.699 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:27.699 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.357636+0000", 2023-10-20T03:36:27.700 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1202e-05 2023-10-20T03:36:27.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.700 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.700 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:27.700 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.367176+0000", 2023-10-20T03:36:27.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0095396060000000008 2023-10-20T03:36:27.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.701 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.701 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.701 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.367223+0000", 2023-10-20T03:36:27.701 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.7113000000000001e-05 2023-10-20T03:36:27.702 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.702 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.702 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.702 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.702 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.702 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=85, reqid=client.4570.0:312, at_version=43'25, trim_to=0'0, roll_forward_to=39'22))", 2023-10-20T03:36:27.703 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:36:27.703 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 62.312853296999997, 2023-10-20T03:36:27.703 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0052771249999999997, 2023-10-20T03:36:27.703 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.703 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:27.704 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.704 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.704 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.704 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:36:27.704 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.705 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.705 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.705 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374352+0000", 2023-10-20T03:36:27.705 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.706 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.706 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.706 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374353+0000", 2023-10-20T03:36:27.706 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6199999999999999e-06 2023-10-20T03:36:27.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.707 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.707 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.707 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374496+0000", 2023-10-20T03:36:27.707 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014282800000000001 2023-10-20T03:36:27.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.708 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.708 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374515+0000", 2023-10-20T03:36:27.708 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8281e-05 2023-10-20T03:36:27.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.709 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.709 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.709 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374527+0000", 2023-10-20T03:36:27.709 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2835e-05 2023-10-20T03:36:27.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.710 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.710 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.710 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374607+0000", 2023-10-20T03:36:27.710 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.9283999999999995e-05 2023-10-20T03:36:27.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.710 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.711 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:27.711 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.374617+0000", 2023-10-20T03:36:27.711 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0394000000000001e-05 2023-10-20T03:36:27.711 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.711 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.711 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:27.712 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.379594+0000", 2023-10-20T03:36:27.712 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049767650000000002 2023-10-20T03:36:27.712 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.712 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.712 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.713 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.379629+0000", 2023-10-20T03:36:27.713 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.5117999999999998e-05 2023-10-20T03:36:27.713 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.713 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.713 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.714 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.714 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.6s4 43/31 ECSubWrite(tid=88, reqid=client.4570.0:313, at_version=43'26, trim_to=0'0, roll_forward_to=43'24))", 2023-10-20T03:36:27.714 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:36:27.715 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 62.299662994999998, 2023-10-20T03:36:27.715 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0049208230000000004, 2023-10-20T03:36:27.715 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.715 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:27.715 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.716 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.716 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.716 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:36:27.716 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.717 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.717 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387542+0000", 2023-10-20T03:36:27.717 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.717 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.718 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.718 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387544+0000", 2023-10-20T03:36:27.718 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6109999999999999e-06 2023-10-20T03:36:27.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.718 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.719 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387771+0000", 2023-10-20T03:36:27.719 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00022766 2023-10-20T03:36:27.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.719 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.719 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.720 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387788+0000", 2023-10-20T03:36:27.720 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7079999999999999e-05 2023-10-20T03:36:27.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.720 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.720 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387799+0000", 2023-10-20T03:36:27.721 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0608000000000001e-05 2023-10-20T03:36:27.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.721 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.721 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.721 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387881+0000", 2023-10-20T03:36:27.722 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2060999999999999e-05 2023-10-20T03:36:27.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.722 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.722 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:27.722 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.387892+0000", 2023-10-20T03:36:27.723 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0912e-05 2023-10-20T03:36:27.723 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.723 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.723 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:27.723 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.392420+0000", 2023-10-20T03:36:27.723 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0045283320000000004 2023-10-20T03:36:27.724 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.724 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.724 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.724 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:25.392463+0000", 2023-10-20T03:36:27.724 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.2558999999999997e-05 2023-10-20T03:36:27.725 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.725 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.725 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.725 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.725 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.725 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:333 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~606208] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:27.726 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:36:27.726 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 61.301275222999998, 2023-10-20T03:36:27.726 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.036441440999999998, 2023-10-20T03:36:27.726 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.726 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:27.727 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:27.727 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:27.727 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:27.727 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 333 2023-10-20T03:36:27.727 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.728 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.728 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.728 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.728 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:36:27.728 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.728 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.729 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.729 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.729 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385930+0000", 2023-10-20T03:36:27.729 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.729 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.730 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.730 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.730 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.385932+0000", 2023-10-20T03:36:27.730 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9929999999999998e-06 2023-10-20T03:36:27.730 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.730 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.731 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.731 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386889+0000", 2023-10-20T03:36:27.731 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000957437 2023-10-20T03:36:27.731 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.731 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.732 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.732 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386904+0000", 2023-10-20T03:36:27.732 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4669e-05 2023-10-20T03:36:27.732 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.732 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.732 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.733 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386910+0000", 2023-10-20T03:36:27.733 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3040000000000001e-06 2023-10-20T03:36:27.733 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.733 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.733 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.734 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.386972+0000", 2023-10-20T03:36:27.734 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.2003999999999998e-05 2023-10-20T03:36:27.734 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.734 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.734 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:27.734 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.387043+0000", 2023-10-20T03:36:27.735 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.0591000000000002e-05 2023-10-20T03:36:27.735 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.735 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.735 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:27.735 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.388080+0000", 2023-10-20T03:36:27.736 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010373190000000001 2023-10-20T03:36:27.736 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.736 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.736 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:27.736 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393723+0000", 2023-10-20T03:36:27.736 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0056426899999999997 2023-10-20T03:36:27.737 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.737 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.737 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:27.737 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422308+0000", 2023-10-20T03:36:27.737 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.028584822999999999 2023-10-20T03:36:27.738 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.738 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.738 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.738 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422371+0000", 2023-10-20T03:36:27.738 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.3610999999999998e-05 2023-10-20T03:36:27.738 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.739 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.739 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.739 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.739 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.739 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:334 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~720896] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:27.740 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:36:27.740 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 61.294140824000003, 2023-10-20T03:36:27.740 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029571147999999998, 2023-10-20T03:36:27.740 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.740 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:27.740 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:27.741 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:27.741 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:27.741 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 334 2023-10-20T03:36:27.741 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.741 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.742 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.742 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.742 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:36:27.742 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.742 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.743 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.743 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.743 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393064+0000", 2023-10-20T03:36:27.743 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.743 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.743 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.744 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.744 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393066+0000", 2023-10-20T03:36:27.744 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.8819999999999999e-06 2023-10-20T03:36:27.744 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.744 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.745 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.745 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393827+0000", 2023-10-20T03:36:27.745 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000760844 2023-10-20T03:36:27.745 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.745 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.745 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.746 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393842+0000", 2023-10-20T03:36:27.746 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5432999999999999e-05 2023-10-20T03:36:27.746 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.746 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.746 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.747 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393850+0000", 2023-10-20T03:36:27.747 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.0199999999999994e-06 2023-10-20T03:36:27.747 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.747 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.747 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.747 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393915+0000", 2023-10-20T03:36:27.748 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 6.4123999999999998e-05 2023-10-20T03:36:27.748 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.748 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.748 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:27.748 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394021+0000", 2023-10-20T03:36:27.748 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010668299999999999 2023-10-20T03:36:27.749 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.749 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.749 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:27.749 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.394832+0000", 2023-10-20T03:36:27.749 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00081066099999999996 2023-10-20T03:36:27.750 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.750 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.750 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:27.750 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405535+0000", 2023-10-20T03:36:27.750 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010703351 2023-10-20T03:36:27.751 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.751 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.751 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:27.751 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422553+0000", 2023-10-20T03:36:27.751 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017018021000000001 2023-10-20T03:36:27.752 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.752 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.752 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.752 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.422635+0000", 2023-10-20T03:36:27.752 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.2128999999999999e-05 2023-10-20T03:36:27.752 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.753 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.753 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.753 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.753 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.753 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:335 3.ds0 3:b8ff490e:::smithi17039300-34:head [append 0~49152] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:27.754 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:36:27.754 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 61.293333848000003, 2023-10-20T03:36:27.754 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.029832813999999999, 2023-10-20T03:36:27.754 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.754 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:27.755 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:27.755 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:27.755 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:27.755 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 335 2023-10-20T03:36:27.755 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.755 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.756 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.756 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.756 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:36:27.756 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.756 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.756 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.757 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.757 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393871+0000", 2023-10-20T03:36:27.757 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.757 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.757 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.758 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.758 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393872+0000", 2023-10-20T03:36:27.758 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6000000000000003e-07 2023-10-20T03:36:27.758 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.758 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.758 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.759 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393919+0000", 2023-10-20T03:36:27.759 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.6576000000000003e-05 2023-10-20T03:36:27.759 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.759 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.759 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.760 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393928+0000", 2023-10-20T03:36:27.760 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1220000000000001e-06 2023-10-20T03:36:27.760 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.760 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.760 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.760 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393937+0000", 2023-10-20T03:36:27.761 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.1740000000000006e-06 2023-10-20T03:36:27.761 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.761 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.761 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.761 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.401885+0000", 2023-10-20T03:36:27.762 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079486149999999992 2023-10-20T03:36:27.762 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.762 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.762 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:27.762 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.402012+0000", 2023-10-20T03:36:27.763 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012668 2023-10-20T03:36:27.763 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.763 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.763 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:27.763 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.403063+0000", 2023-10-20T03:36:27.764 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0010508410000000001 2023-10-20T03:36:27.764 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.764 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.764 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:27.764 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405538+0000", 2023-10-20T03:36:27.764 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0024754519999999999 2023-10-20T03:36:27.765 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.765 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.765 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:27.765 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.423630+0000", 2023-10-20T03:36:27.765 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.018091716000000001 2023-10-20T03:36:27.766 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.766 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.766 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.766 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.423704+0000", 2023-10-20T03:36:27.766 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.3878e-05 2023-10-20T03:36:27.767 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.767 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.767 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.767 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.767 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.767 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:336 3.ds0 3:b8ff490e:::smithi17039300-34:head [setxattr _header (56) in=7b] snapc b=[b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:27.768 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:36:27.768 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 61.293254472999998, 2023-10-20T03:36:27.768 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.030244758, 2023-10-20T03:36:27.768 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.768 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:27.769 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:27.769 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:27.769 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:27.769 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 336 2023-10-20T03:36:27.769 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.769 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.770 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.770 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.770 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:36:27.770 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.770 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.771 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.771 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.771 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:36:27.771 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.771 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.771 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.772 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.773 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393951+0000", 2023-10-20T03:36:27.773 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.8400000000000004e-07 2023-10-20T03:36:27.773 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.774 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.774 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.774 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393955+0000", 2023-10-20T03:36:27.774 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.8210000000000003e-06 2023-10-20T03:36:27.774 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.775 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.775 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.775 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393963+0000", 2023-10-20T03:36:27.775 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.8359999999999992e-06 2023-10-20T03:36:27.775 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.776 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.776 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.776 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393968+0000", 2023-10-20T03:36:27.776 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.3689999999999998e-06 2023-10-20T03:36:27.776 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.776 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.777 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.777 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404110+0000", 2023-10-20T03:36:27.777 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.010141582 2023-10-20T03:36:27.777 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.777 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.777 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:27.778 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.404246+0000", 2023-10-20T03:36:27.778 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00013631799999999999 2023-10-20T03:36:27.778 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.778 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.778 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:27.779 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405009+0000", 2023-10-20T03:36:27.779 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00076302800000000001 2023-10-20T03:36:27.779 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.779 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.779 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:27.779 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.406198+0000", 2023-10-20T03:36:27.780 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0011889610000000001 2023-10-20T03:36:27.780 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.780 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.780 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:27.780 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424049+0000", 2023-10-20T03:36:27.781 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.017850957000000001 2023-10-20T03:36:27.781 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.781 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.781 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.781 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424195+0000", 2023-10-20T03:36:27.781 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014630200000000001 2023-10-20T03:36:27.782 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.782 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.782 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.782 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.782 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.783 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:337 3.ds0 3:b8ff490e:::smithi17039300-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-10-20T03:36:27.783 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:36:27.783 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 61.293227688999998, 2023-10-20T03:36:27.783 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.032645448000000001, 2023-10-20T03:36:27.784 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.784 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:36:27.784 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:27.784 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:27.784 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:27.785 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 337 2023-10-20T03:36:27.785 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.785 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.785 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.785 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.785 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:36:27.786 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.786 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.786 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.786 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.786 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393977+0000", 2023-10-20T03:36:27.787 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.787 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.787 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.787 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393978+0000", 2023-10-20T03:36:27.787 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0800000000000005e-07 2023-10-20T03:36:27.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.788 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.788 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.788 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393981+0000", 2023-10-20T03:36:27.788 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.7199999999999998e-06 2023-10-20T03:36:27.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.789 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.789 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.789 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393988+0000", 2023-10-20T03:36:27.789 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.5939999999999996e-06 2023-10-20T03:36:27.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.790 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.790 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.790 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.393993+0000", 2023-10-20T03:36:27.790 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.0200000000000002e-06 2023-10-20T03:36:27.790 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.791 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.791 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405399+0000", 2023-10-20T03:36:27.791 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.011406229 2023-10-20T03:36:27.791 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.791 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.792 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:27.792 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.405522+0000", 2023-10-20T03:36:27.792 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00012226199999999999 2023-10-20T03:36:27.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.792 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.793 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.793 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424833+0000", 2023-10-20T03:36:27.793 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.019311524 2023-10-20T03:36:27.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.793 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:27.794 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.424976+0000", 2023-10-20T03:36:27.794 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00014297600000000001 2023-10-20T03:36:27.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.794 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.794 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.795 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:26.426623+0000", 2023-10-20T03:36:27.795 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0016466149999999999 2023-10-20T03:36:27.795 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.795 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.795 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.796 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.796 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:384 3.ds0 3:ba203010:::smithi17039300-4:head [copy-from ver 21] snapc f=[f,d,b,a,9,6,3,2] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:36:27.796 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:36:27.796 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 57.316444406999999, 2023-10-20T03:36:27.796 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.077402023, 2023-10-20T03:36:27.797 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.797 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:27.797 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:27.797 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:27.797 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:27.797 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 384 2023-10-20T03:36:27.798 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.798 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.798 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.798 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.798 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:36:27.799 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.799 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.799 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.799 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370761+0000", 2023-10-20T03:36:27.799 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.800 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.800 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.800 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.800 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370763+0000", 2023-10-20T03:36:27.800 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4569999999999999e-06 2023-10-20T03:36:27.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.801 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.801 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.801 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370776+0000", 2023-10-20T03:36:27.801 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2483000000000001e-05 2023-10-20T03:36:27.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.802 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.802 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.802 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370809+0000", 2023-10-20T03:36:27.802 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2916000000000001e-05 2023-10-20T03:36:27.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.803 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.803 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370826+0000", 2023-10-20T03:36:27.803 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7183000000000001e-05 2023-10-20T03:36:27.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.803 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.804 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.804 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371075+0000", 2023-10-20T03:36:27.804 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.000249387 2023-10-20T03:36:27.804 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.804 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.805 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:27.805 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371389+0000", 2023-10-20T03:36:27.805 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00031429899999999998 2023-10-20T03:36:27.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.805 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.806 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:27.806 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.392222+0000", 2023-10-20T03:36:27.806 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.020832532000000001 2023-10-20T03:36:27.806 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.806 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.806 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:27.807 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.447199+0000", 2023-10-20T03:36:27.807 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.054976917 2023-10-20T03:36:27.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.807 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.807 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "commit_sent", 2023-10-20T03:36:27.808 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.448009+0000", 2023-10-20T03:36:27.808 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00080978099999999998 2023-10-20T03:36:27.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.808 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.808 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.808 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.448163+0000", 2023-10-20T03:36:27.809 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015406799999999999 2023-10-20T03:36:27.809 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.809 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.809 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.810 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.810 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(client.4570.0:385 3.ds0 3:ba203010:::smithi17039300-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:36:27.810 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:36:27.810 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 57.316344282999999, 2023-10-20T03:36:27.810 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.074370642000000001, 2023-10-20T03:36:27.811 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.811 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:36:27.811 INFO:teuthology.orchestra.run.smithi033.stdout: "client_info": { 2023-10-20T03:36:27.811 INFO:teuthology.orchestra.run.smithi033.stdout: "client": "client.4570", 2023-10-20T03:36:27.811 INFO:teuthology.orchestra.run.smithi033.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:27.811 INFO:teuthology.orchestra.run.smithi033.stdout: "tid": 385 2023-10-20T03:36:27.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.812 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.812 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.812 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.812 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:36:27.812 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.813 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.813 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370861+0000", 2023-10-20T03:36:27.813 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.814 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.814 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.814 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.814 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370862+0000", 2023-10-20T03:36:27.814 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.2699999999999998e-07 2023-10-20T03:36:27.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.815 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.815 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.815 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370866+0000", 2023-10-20T03:36:27.815 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 4.42e-06 2023-10-20T03:36:27.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.816 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.816 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370880+0000", 2023-10-20T03:36:27.816 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3833e-05 2023-10-20T03:36:27.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.816 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.817 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.817 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.370889+0000", 2023-10-20T03:36:27.817 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 8.5900000000000008e-06 2023-10-20T03:36:27.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.817 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.818 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.818 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.371887+0000", 2023-10-20T03:36:27.818 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00099814499999999998 2023-10-20T03:36:27.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.819 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "waiting for blocked object", 2023-10-20T03:36:27.819 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.372069+0000", 2023-10-20T03:36:27.819 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00018220899999999999 2023-10-20T03:36:27.819 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.819 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.819 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.820 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445044+0000", 2023-10-20T03:36:27.820 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.072974686999999996 2023-10-20T03:36:27.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.820 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.820 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:27.821 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445137+0000", 2023-10-20T03:36:27.821 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.3694000000000001e-05 2023-10-20T03:36:27.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.821 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.821 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.821 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445231+0000", 2023-10-20T03:36:27.822 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4136999999999997e-05 2023-10-20T03:36:27.822 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.822 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.822 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.823 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.823 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "osd_op(osd.12.22:4 3.ds0 3:b8ff490e:::smithi17039300-34:head [assert-version v21,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2023-10-20T03:36:27.823 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:36:27.823 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 57.313895559999999, 2023-10-20T03:36:27.823 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0083526739999999992, 2023-10-20T03:36:27.823 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.824 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "started", 2023-10-20T03:36:27.824 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.824 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.824 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.824 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:36:27.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.825 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.825 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.825 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373310+0000", 2023-10-20T03:36:27.825 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.826 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.826 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.826 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373312+0000", 2023-10-20T03:36:27.826 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.424e-06 2023-10-20T03:36:27.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.827 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.827 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.827 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373326+0000", 2023-10-20T03:36:27.827 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4226000000000001e-05 2023-10-20T03:36:27.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.828 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.828 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.828 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373357+0000", 2023-10-20T03:36:27.828 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0698999999999999e-05 2023-10-20T03:36:27.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.828 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.829 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.829 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373388+0000", 2023-10-20T03:36:27.829 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0846000000000002e-05 2023-10-20T03:36:27.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.829 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.830 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373551+0000", 2023-10-20T03:36:27.830 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00016350300000000001 2023-10-20T03:36:27.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.830 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.830 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "started", 2023-10-20T03:36:27.831 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.373703+0000", 2023-10-20T03:36:27.831 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015216200000000001 2023-10-20T03:36:27.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.831 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.831 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.832 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.381662+0000", 2023-10-20T03:36:27.832 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0079588139999999998 2023-10-20T03:36:27.832 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.832 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.832 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.832 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.833 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.833 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpReadReply(3.ds0 48/31 ECSubReadReply(tid=59, attrs_read=0))", 2023-10-20T03:36:27.833 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:36:27.833 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 57.310963041999997, 2023-10-20T03:36:27.833 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0055763890000000002, 2023-10-20T03:36:27.833 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.834 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:27.834 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.834 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.834 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.834 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:36:27.835 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.835 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.835 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.835 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376242+0000", 2023-10-20T03:36:27.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.836 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.836 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.836 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.376244+0000", 2023-10-20T03:36:27.836 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7570000000000001e-06 2023-10-20T03:36:27.837 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.837 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.837 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.837 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380643+0000", 2023-10-20T03:36:27.837 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00439931 2023-10-20T03:36:27.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.838 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.838 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380671+0000", 2023-10-20T03:36:27.838 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.8019000000000001e-05 2023-10-20T03:36:27.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.839 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.839 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.839 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380685+0000", 2023-10-20T03:36:27.839 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4334e-05 2023-10-20T03:36:27.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.840 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.840 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.380780+0000", 2023-10-20T03:36:27.840 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4087999999999998e-05 2023-10-20T03:36:27.840 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.840 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.841 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.841 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.381818+0000", 2023-10-20T03:36:27.841 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.001038881 2023-10-20T03:36:27.841 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.841 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.842 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.842 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.842 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs2 48/47 ECSubWrite(tid=115, reqid=osd.13.0:115, at_version=48'27, trim_to=0'0, roll_forward_to=0'0))", 2023-10-20T03:36:27.842 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:36:27.843 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 57.297665170000002, 2023-10-20T03:36:27.843 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0047760049999999998, 2023-10-20T03:36:27.843 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.843 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:27.843 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.843 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.844 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.844 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:36:27.844 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.844 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.845 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.845 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389540+0000", 2023-10-20T03:36:27.845 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.845 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.846 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389541+0000", 2023-10-20T03:36:27.846 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5659999999999999e-06 2023-10-20T03:36:27.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.846 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.846 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.847 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389579+0000", 2023-10-20T03:36:27.847 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7070999999999998e-05 2023-10-20T03:36:27.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.847 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.847 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389605+0000", 2023-10-20T03:36:27.848 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6848000000000001e-05 2023-10-20T03:36:27.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.848 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.848 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.848 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389626+0000", 2023-10-20T03:36:27.849 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0579000000000001e-05 2023-10-20T03:36:27.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.849 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.849 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.849 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389781+0000", 2023-10-20T03:36:27.850 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015502300000000001 2023-10-20T03:36:27.850 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.850 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.850 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:27.850 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.389807+0000", 2023-10-20T03:36:27.850 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6418999999999999e-05 2023-10-20T03:36:27.851 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.851 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.851 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:27.851 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394222+0000", 2023-10-20T03:36:27.851 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044143139999999999 2023-10-20T03:36:27.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.852 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.852 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.394316+0000", 2023-10-20T03:36:27.852 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 9.4184999999999994e-05 2023-10-20T03:36:27.852 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.853 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.853 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.853 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.853 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:27.853 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:36:27.854 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 57.257790057999998, 2023-10-20T03:36:27.854 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015918188, 2023-10-20T03:36:27.854 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.854 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:27.854 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.855 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.855 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:36:27.855 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.855 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.856 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.856 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429415+0000", 2023-10-20T03:36:27.856 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.856 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.857 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429417+0000", 2023-10-20T03:36:27.857 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9429999999999999e-06 2023-10-20T03:36:27.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.857 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.857 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.858 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429432+0000", 2023-10-20T03:36:27.858 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4942e-05 2023-10-20T03:36:27.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.858 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.858 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.859 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429459+0000", 2023-10-20T03:36:27.859 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.6740000000000001e-05 2023-10-20T03:36:27.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.859 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.859 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.859 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429472+0000", 2023-10-20T03:36:27.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3382e-05 2023-10-20T03:36:27.860 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.860 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.860 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.860 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445294+0000", 2023-10-20T03:36:27.860 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015821894999999999 2023-10-20T03:36:27.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.861 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.861 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.861 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445333+0000", 2023-10-20T03:36:27.861 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9286000000000003e-05 2023-10-20T03:36:27.862 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.862 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.862 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.862 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.862 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:27.863 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:36:27.863 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 57.257441075000003, 2023-10-20T03:36:27.863 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01562377, 2023-10-20T03:36:27.863 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.863 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:27.864 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.864 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.864 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.864 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:36:27.864 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.865 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.865 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.865 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429764+0000", 2023-10-20T03:36:27.865 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.866 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.866 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429765+0000", 2023-10-20T03:36:27.866 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0079999999999999e-06 2023-10-20T03:36:27.866 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.866 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.867 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429772+0000", 2023-10-20T03:36:27.867 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.2230000000000001e-06 2023-10-20T03:36:27.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.867 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.867 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.868 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429793+0000", 2023-10-20T03:36:27.868 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0707999999999999e-05 2023-10-20T03:36:27.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.868 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.868 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.429806+0000", 2023-10-20T03:36:27.869 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3528e-05 2023-10-20T03:36:27.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.869 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.869 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.869 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445370+0000", 2023-10-20T03:36:27.870 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.015563917 2023-10-20T03:36:27.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.870 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.870 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.870 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445388+0000", 2023-10-20T03:36:27.871 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.7385999999999999e-05 2023-10-20T03:36:27.871 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.871 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.871 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.872 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:27.872 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:36:27.872 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 57.255790179000002, 2023-10-20T03:36:27.872 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.014045581999999999, 2023-10-20T03:36:27.872 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.873 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:27.873 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.873 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.873 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.873 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:36:27.873 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.874 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.874 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.874 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431415+0000", 2023-10-20T03:36:27.874 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.875 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.875 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.875 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431417+0000", 2023-10-20T03:36:27.875 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.4159999999999998e-06 2023-10-20T03:36:27.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.876 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.876 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431433+0000", 2023-10-20T03:36:27.876 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.5653000000000001e-05 2023-10-20T03:36:27.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.876 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.877 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.877 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431472+0000", 2023-10-20T03:36:27.877 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.9232e-05 2023-10-20T03:36:27.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.877 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.878 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.431492+0000", 2023-10-20T03:36:27.878 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9550000000000001e-05 2023-10-20T03:36:27.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.878 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.878 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.879 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445435+0000", 2023-10-20T03:36:27.879 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.01394307 2023-10-20T03:36:27.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.879 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.879 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.880 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445460+0000", 2023-10-20T03:36:27.880 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.5661e-05 2023-10-20T03:36:27.880 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.880 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.880 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.881 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.881 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:27.881 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:36:27.881 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 57.254984178999997, 2023-10-20T03:36:27.881 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013263258999999999, 2023-10-20T03:36:27.881 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.882 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:27.882 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.882 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.882 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.882 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:36:27.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.883 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.883 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.883 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432221+0000", 2023-10-20T03:36:27.883 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.884 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.884 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.884 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432228+0000", 2023-10-20T03:36:27.884 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.08e-06 2023-10-20T03:36:27.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.885 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.885 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.885 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432236+0000", 2023-10-20T03:36:27.885 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 7.6629999999999993e-06 2023-10-20T03:36:27.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.886 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.886 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.886 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432260+0000", 2023-10-20T03:36:27.886 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.3998999999999998e-05 2023-10-20T03:36:27.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.887 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.887 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432272+0000", 2023-10-20T03:36:27.887 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.2023e-05 2023-10-20T03:36:27.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.887 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.888 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.888 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445470+0000", 2023-10-20T03:36:27.888 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.013198507 2023-10-20T03:36:27.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.888 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.889 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.889 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445484+0000", 2023-10-20T03:36:27.889 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.3987e-05 2023-10-20T03:36:27.889 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.889 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.889 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.890 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.890 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 48/31 ECSubWriteReply(tid=60, last_complete=48'23, committed=1, applied=1))", 2023-10-20T03:36:27.890 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:36:27.890 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 57.254679830000001, 2023-10-20T03:36:27.891 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012997794, 2023-10-20T03:36:27.891 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.891 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:27.891 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.891 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.891 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.892 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:36:27.892 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.892 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.892 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.892 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432525+0000", 2023-10-20T03:36:27.893 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.893 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.893 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.893 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432526+0000", 2023-10-20T03:36:27.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.1400000000000001e-06 2023-10-20T03:36:27.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.894 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.894 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.894 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432532+0000", 2023-10-20T03:36:27.894 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 5.823e-06 2023-10-20T03:36:27.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.895 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.895 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.895 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432552+0000", 2023-10-20T03:36:27.895 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0032999999999999e-05 2023-10-20T03:36:27.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.896 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.896 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.896 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.432567+0000", 2023-10-20T03:36:27.896 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.4932999999999999e-05 2023-10-20T03:36:27.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.897 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.897 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445506+0000", 2023-10-20T03:36:27.897 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.012939315 2023-10-20T03:36:27.897 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.897 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.898 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.898 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:30.445523+0000", 2023-10-20T03:36:27.898 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.6549999999999999e-05 2023-10-20T03:36:27.898 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.898 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.899 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.899 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.899 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpWrite(3.cs2 49/47 ECSubWrite(tid=118, reqid=osd.13.0:118, at_version=49'29, trim_to=0'0, roll_forward_to=48'27))", 2023-10-20T03:36:27.899 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:36:27.899 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 56.276346599, 2023-10-20T03:36:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0043378410000000003, 2023-10-20T03:36:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.900 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.901 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.901 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:36:27.901 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.901 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.901 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.902 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410858+0000", 2023-10-20T03:36:27.902 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.902 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.902 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.902 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.903 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410859+0000", 2023-10-20T03:36:27.903 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.0699999999999999e-07 2023-10-20T03:36:27.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.903 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.903 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.903 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410884+0000", 2023-10-20T03:36:27.904 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 2.529e-05 2023-10-20T03:36:27.904 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.904 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.904 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.904 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410898+0000", 2023-10-20T03:36:27.904 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.383e-05 2023-10-20T03:36:27.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.905 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.905 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.905 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.410928+0000", 2023-10-20T03:36:27.905 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.0329e-05 2023-10-20T03:36:27.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.906 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.906 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.906 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.411037+0000", 2023-10-20T03:36:27.906 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00010869099999999999 2023-10-20T03:36:27.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.907 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.907 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_started", 2023-10-20T03:36:27.907 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.411048+0000", 2023-10-20T03:36:27.907 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.0905e-05 2023-10-20T03:36:27.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.908 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.908 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "sub_op_committed", 2023-10-20T03:36:27.908 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415159+0000", 2023-10-20T03:36:27.908 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0041115520000000001 2023-10-20T03:36:27.908 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.908 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.909 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.909 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:31.415196+0000", 2023-10-20T03:36:27.909 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.7036999999999998e-05 2023-10-20T03:36:27.909 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.909 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.909 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.910 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.910 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.910 INFO:teuthology.orchestra.run.smithi033.stdout: "description": "MOSDECSubOpRead(3.cs2 55/54 ECSubRead(tid=70, to_read={3:3382ea17:::smithi17039300-35:6=0,262144,0}, subchunks={3:3382ea17:::smithi17039300-35:6=[0,1]}, attrs_to_read=))", 2023-10-20T03:36:27.910 INFO:teuthology.orchestra.run.smithi033.stdout: "initiated_at": "2023-10-20T03:35:37.459240+0000", 2023-10-20T03:36:27.910 INFO:teuthology.orchestra.run.smithi033.stdout: "age": 50.227965492999999, 2023-10-20T03:36:27.911 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0044608460000000001, 2023-10-20T03:36:27.911 INFO:teuthology.orchestra.run.smithi033.stdout: "type_data": { 2023-10-20T03:36:27.911 INFO:teuthology.orchestra.run.smithi033.stdout: "flag_point": "reached pg", 2023-10-20T03:36:27.911 INFO:teuthology.orchestra.run.smithi033.stdout: "events": [ 2023-10-20T03:36:27.911 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.911 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "initiated", 2023-10-20T03:36:27.912 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459240+0000", 2023-10-20T03:36:27.912 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.912 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.912 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "header_read", 2023-10-20T03:36:27.913 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459240+0000", 2023-10-20T03:36:27.913 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0 2023-10-20T03:36:27.913 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.913 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.913 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "throttled", 2023-10-20T03:36:27.913 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459241+0000", 2023-10-20T03:36:27.914 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.84e-06 2023-10-20T03:36:27.914 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.914 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.914 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "all_read", 2023-10-20T03:36:27.914 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459256+0000", 2023-10-20T03:36:27.914 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.454e-05 2023-10-20T03:36:27.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.915 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.915 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "dispatched", 2023-10-20T03:36:27.915 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459288+0000", 2023-10-20T03:36:27.915 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 3.2329000000000002e-05 2023-10-20T03:36:27.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.916 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.916 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "queued_for_pg", 2023-10-20T03:36:27.916 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459307+0000", 2023-10-20T03:36:27.916 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 1.9046999999999999e-05 2023-10-20T03:36:27.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.917 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.917 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "reached_pg", 2023-10-20T03:36:27.917 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.459460+0000", 2023-10-20T03:36:27.917 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.00015253200000000001 2023-10-20T03:36:27.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-10-20T03:36:27.918 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-10-20T03:36:27.918 INFO:teuthology.orchestra.run.smithi033.stdout: "event": "done", 2023-10-20T03:36:27.918 INFO:teuthology.orchestra.run.smithi033.stdout: "time": "2023-10-20T03:35:37.463700+0000", 2023-10-20T03:36:27.918 INFO:teuthology.orchestra.run.smithi033.stdout: "duration": 0.0042405580000000002 2023-10-20T03:36:27.918 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.918 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.919 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.919 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-10-20T03:36:27.919 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2023-10-20T03:36:27.919 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-10-20T03:36:27.921 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-10-20T03:36:27.921 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:27.920+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:28.022 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:28.019+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:28.057 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:28.058 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:36:28.058 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:36:28.058 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:28.072 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-10-20T03:36:28.123 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:28.120+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:28.189 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:28.190 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:36:28.190 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:36:28.190 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:36:28.190 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:28.204 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-10-20T03:36:28.224 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:28.220+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:28.297 INFO:tasks.rados.rados.0.smithi170.stdout:394: done (0 left) 2023-10-20T03:36:28.297 INFO:tasks.rados.rados.0.smithi170.stdout:append oid 19 current snap is 35 2023-10-20T03:36:28.297 INFO:tasks.rados.rados.0.smithi170.stdout:395: seq_num 156 ranges {3686400=229376} 2023-10-20T03:36:28.325 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:36:28.320+0000 7f9e54b37640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:36:28.336 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:28.336 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:36:28.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:36:28.336 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2023-10-20T03:36:28.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.337 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.bs1 41/31 ECSubWrite(tid=77, reqid=client.4570.0:293, at_version=41'10, trim_to=0'0, roll_forward_to=39'8))", 2023-10-20T03:36:28.337 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:23.461353+0000", 2023-10-20T03:36:28.337 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.871358526999998, 2023-10-20T03:36:28.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018876751000000001, 2023-10-20T03:36:28.338 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.338 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:28.338 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.339 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.461353+0000", 2023-10-20T03:36:28.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.461353+0000", 2023-10-20T03:36:28.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.461354+0000", 2023-10-20T03:36:28.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.029e-06 2023-10-20T03:36:28.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462244+0000", 2023-10-20T03:36:28.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089060400000000003 2023-10-20T03:36:28.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.343 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462291+0000", 2023-10-20T03:36:28.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6476e-05 2023-10-20T03:36:28.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.343 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462316+0000", 2023-10-20T03:36:28.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5320000000000002e-05 2023-10-20T03:36:28.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462615+0000", 2023-10-20T03:36:28.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00029899800000000003 2023-10-20T03:36:28.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:28.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.462645+0000", 2023-10-20T03:36:28.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0352999999999999e-05 2023-10-20T03:36:28.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:28.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.480122+0000", 2023-10-20T03:36:28.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017476588000000001 2023-10-20T03:36:28.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.348 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:23.480229+0000", 2023-10-20T03:36:28.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107383 2023-10-20T03:36:28.348 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.348 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.348 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.349 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.7s3 60/31 ECSubWrite(tid=200, reqid=client.4570.0:459, at_version=60'9, trim_to=0'0, roll_forward_to=38'7))", 2023-10-20T03:36:28.349 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:41.899303+0000", 2023-10-20T03:36:28.349 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 46.433407836999997, 2023-10-20T03:36:28.349 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.023677786999999999, 2023-10-20T03:36:28.350 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.350 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:28.350 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.351 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899303+0000", 2023-10-20T03:36:28.351 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.351 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.352 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899303+0000", 2023-10-20T03:36:28.352 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.352 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899305+0000", 2023-10-20T03:36:28.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.813e-06 2023-10-20T03:36:28.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.353 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899602+0000", 2023-10-20T03:36:28.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00029634300000000002 2023-10-20T03:36:28.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899630+0000", 2023-10-20T03:36:28.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8085999999999998e-05 2023-10-20T03:36:28.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.356 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899663+0000", 2023-10-20T03:36:28.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3648e-05 2023-10-20T03:36:28.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.356 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899783+0000", 2023-10-20T03:36:28.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119461 2023-10-20T03:36:28.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:28.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.899803+0000", 2023-10-20T03:36:28.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0335000000000001e-05 2023-10-20T03:36:28.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:28.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.922864+0000", 2023-10-20T03:36:28.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.023061324000000001 2023-10-20T03:36:28.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:41.922981+0000", 2023-10-20T03:36:28.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000116777 2023-10-20T03:36:28.360 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.360 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.360 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.361 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:545 3.8s1 3:1166f250:::smithi17039300-32:head [read 0~3375104,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2023-10-20T03:36:28.361 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:46.883962+0000", 2023-10-20T03:36:28.361 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.448748960000003, 2023-10-20T03:36:28.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025717769000000001, 2023-10-20T03:36:28.362 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.362 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:28.362 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:28.362 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:28.362 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:28.363 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 545 2023-10-20T03:36:28.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.363 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.883962+0000", 2023-10-20T03:36:28.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.883962+0000", 2023-10-20T03:36:28.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.883967+0000", 2023-10-20T03:36:28.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.284e-06 2023-10-20T03:36:28.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.366 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.883991+0000", 2023-10-20T03:36:28.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4093000000000001e-05 2023-10-20T03:36:28.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884042+0000", 2023-10-20T03:36:28.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1819e-05 2023-10-20T03:36:28.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884070+0000", 2023-10-20T03:36:28.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.758e-05 2023-10-20T03:36:28.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884323+0000", 2023-10-20T03:36:28.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025264500000000002 2023-10-20T03:36:28.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op must wait for map", 2023-10-20T03:36:28.370 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884348+0000", 2023-10-20T03:36:28.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5276e-05 2023-10-20T03:36:28.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.889737+0000", 2023-10-20T03:36:28.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053888989999999999 2023-10-20T03:36:28.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:28.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.890003+0000", 2023-10-20T03:36:28.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026633800000000002 2023-10-20T03:36:28.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.909680+0000", 2023-10-20T03:36:28.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019676835 2023-10-20T03:36:28.373 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.374 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.374 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.374 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:546 3.8s1 3:1166f250:::smithi17039300-32:head [read 0~3375104,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2023-10-20T03:36:28.375 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:46.884128+0000", 2023-10-20T03:36:28.375 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.448583047, 2023-10-20T03:36:28.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031700702999999997, 2023-10-20T03:36:28.375 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.375 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:28.376 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:28.376 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:28.376 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:28.376 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 546 2023-10-20T03:36:28.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.377 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884128+0000", 2023-10-20T03:36:28.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884128+0000", 2023-10-20T03:36:28.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884130+0000", 2023-10-20T03:36:28.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.536e-06 2023-10-20T03:36:28.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884140+0000", 2023-10-20T03:36:28.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0186999999999999e-05 2023-10-20T03:36:28.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884165+0000", 2023-10-20T03:36:28.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5276e-05 2023-10-20T03:36:28.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884183+0000", 2023-10-20T03:36:28.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7496999999999999e-05 2023-10-20T03:36:28.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884489+0000", 2023-10-20T03:36:28.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030652099999999997 2023-10-20T03:36:28.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:28.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884510+0000", 2023-10-20T03:36:28.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.088e-05 2023-10-20T03:36:28.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.895027+0000", 2023-10-20T03:36:28.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010516635999999999 2023-10-20T03:36:28.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:28.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.895341+0000", 2023-10-20T03:36:28.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00031437799999999998 2023-10-20T03:36:28.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.915829+0000", 2023-10-20T03:36:28.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020487792000000001 2023-10-20T03:36:28.387 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.388 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.388 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.388 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:547 3.8s1 3:1166f250:::smithi17039300-32:head [read 0~3375104,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2023-10-20T03:36:28.389 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:46.884211+0000", 2023-10-20T03:36:28.389 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.448500301999999, 2023-10-20T03:36:28.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.037539995999999999, 2023-10-20T03:36:28.389 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.389 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:28.390 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:28.390 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:28.390 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:28.390 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 547 2023-10-20T03:36:28.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.391 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884211+0000", 2023-10-20T03:36:28.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884211+0000", 2023-10-20T03:36:28.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884212+0000", 2023-10-20T03:36:28.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0669999999999999e-06 2023-10-20T03:36:28.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884218+0000", 2023-10-20T03:36:28.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9699999999999996e-06 2023-10-20T03:36:28.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884262+0000", 2023-10-20T03:36:28.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3949999999999998e-05 2023-10-20T03:36:28.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884280+0000", 2023-10-20T03:36:28.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7632999999999999e-05 2023-10-20T03:36:28.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884611+0000", 2023-10-20T03:36:28.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00033177400000000001 2023-10-20T03:36:28.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:28.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.884634+0000", 2023-10-20T03:36:28.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2909999999999999e-05 2023-10-20T03:36:28.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.897834+0000", 2023-10-20T03:36:28.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013200197 2023-10-20T03:36:28.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:28.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.898033+0000", 2023-10-20T03:36:28.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019837500000000001 2023-10-20T03:36:28.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.921751+0000", 2023-10-20T03:36:28.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.023718119999999999 2023-10-20T03:36:28.401 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.401 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.402 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.402 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpReadReply(3.8s1 65/31 ECSubReadReply(tid=50, attrs_read=0))", 2023-10-20T03:36:28.402 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:46.903170+0000", 2023-10-20T03:36:28.403 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.429540877999997, 2023-10-20T03:36:28.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018730475, 2023-10-20T03:36:28.403 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.403 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:28.403 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.903170+0000", 2023-10-20T03:36:28.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.903170+0000", 2023-10-20T03:36:28.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.903171+0000", 2023-10-20T03:36:28.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.113e-06 2023-10-20T03:36:28.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.904062+0000", 2023-10-20T03:36:28.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089037999999999997 2023-10-20T03:36:28.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.904081+0000", 2023-10-20T03:36:28.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8766999999999998e-05 2023-10-20T03:36:28.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.904091+0000", 2023-10-20T03:36:28.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813e-05 2023-10-20T03:36:28.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.916506+0000", 2023-10-20T03:36:28.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012414236 2023-10-20T03:36:28.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:46.921901+0000", 2023-10-20T03:36:28.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053951659999999999 2023-10-20T03:36:28.411 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.411 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.411 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.412 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.3s2 71/56 ECSubWrite(tid=63, reqid=client.4570.0:575, at_version=71'46, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:28.412 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:52.983764+0000", 2023-10-20T03:36:28.412 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 35.348947090000003, 2023-10-20T03:36:28.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018745662, 2023-10-20T03:36:28.413 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.413 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:28.413 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.983764+0000", 2023-10-20T03:36:28.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.983764+0000", 2023-10-20T03:36:28.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.983766+0000", 2023-10-20T03:36:28.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.657e-06 2023-10-20T03:36:28.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.984152+0000", 2023-10-20T03:36:28.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038608000000000002 2023-10-20T03:36:28.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.984182+0000", 2023-10-20T03:36:28.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0360000000000001e-05 2023-10-20T03:36:28.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.984198+0000", 2023-10-20T03:36:28.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5290000000000001e-05 2023-10-20T03:36:28.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.984327+0000", 2023-10-20T03:36:28.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129066 2023-10-20T03:36:28.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:28.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:52.984347+0000", 2023-10-20T03:36:28.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0186999999999998e-05 2023-10-20T03:36:28.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:28.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.002400+0000", 2023-10-20T03:36:28.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018053205999999999 2023-10-20T03:36:28.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.002510+0000", 2023-10-20T03:36:28.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010981599999999999 2023-10-20T03:36:28.423 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.423 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.423 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.424 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.3s2 71/56 ECSubWrite(tid=64, reqid=client.4570.0:576, at_version=71'47, trim_to=0'0, roll_forward_to=71'45))", 2023-10-20T03:36:28.424 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.006326+0000", 2023-10-20T03:36:28.424 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 35.326385510999998, 2023-10-20T03:36:28.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030838075, 2023-10-20T03:36:28.425 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.425 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:28.425 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006326+0000", 2023-10-20T03:36:28.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006326+0000", 2023-10-20T03:36:28.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006328+0000", 2023-10-20T03:36:28.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2500000000000001e-06 2023-10-20T03:36:28.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006578+0000", 2023-10-20T03:36:28.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024969899999999998 2023-10-20T03:36:28.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006610+0000", 2023-10-20T03:36:28.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2499000000000002e-05 2023-10-20T03:36:28.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006627+0000", 2023-10-20T03:36:28.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6487e-05 2023-10-20T03:36:28.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006800+0000", 2023-10-20T03:36:28.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173439 2023-10-20T03:36:28.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:28.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.006834+0000", 2023-10-20T03:36:28.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469000000000001e-05 2023-10-20T03:36:28.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:28.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.037070+0000", 2023-10-20T03:36:28.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030235952999999999 2023-10-20T03:36:28.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.037164+0000", 2023-10-20T03:36:28.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4278999999999996e-05 2023-10-20T03:36:28.435 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.435 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.435 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.436 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:611 3.ds0 3:b8ff490e:::smithi17039300-34:head [stat,delete] snapc 28=[28,27,26] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-10-20T03:36:28.436 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:55.989324+0000", 2023-10-20T03:36:28.436 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 32.343387491000001, 2023-10-20T03:36:28.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021838554, 2023-10-20T03:36:28.437 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.437 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:28.437 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:28.437 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:28.437 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:28.437 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 611 2023-10-20T03:36:28.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.438 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989324+0000", 2023-10-20T03:36:28.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989324+0000", 2023-10-20T03:36:28.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989327+0000", 2023-10-20T03:36:28.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.96e-06 2023-10-20T03:36:28.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989339+0000", 2023-10-20T03:36:28.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.261e-05 2023-10-20T03:36:28.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989376+0000", 2023-10-20T03:36:28.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6976e-05 2023-10-20T03:36:28.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989406+0000", 2023-10-20T03:36:28.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9873e-05 2023-10-20T03:36:28.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989561+0000", 2023-10-20T03:36:28.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015517300000000001 2023-10-20T03:36:28.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op must wait for map", 2023-10-20T03:36:28.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:55.989579+0000", 2023-10-20T03:36:28.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7846000000000002e-05 2023-10-20T03:36:28.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.000958+0000", 2023-10-20T03:36:28.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01137907 2023-10-20T03:36:28.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:28.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.001262+0000", 2023-10-20T03:36:28.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030417399999999999 2023-10-20T03:36:28.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:28.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.002906+0000", 2023-10-20T03:36:28.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001643191 2023-10-20T03:36:28.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:28.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.007450+0000", 2023-10-20T03:36:28.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045443389999999997 2023-10-20T03:36:28.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:36:28.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.011024+0000", 2023-10-20T03:36:28.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035745400000000002 2023-10-20T03:36:28.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:56.011162+0000", 2023-10-20T03:36:28.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013780199999999999 2023-10-20T03:36:28.451 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.452 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.452 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.453 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:619 3.ds0 3:bbaca854:::smithi17039300-44:head [read 0~737280,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2023-10-20T03:36:28.453 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:57.009377+0000", 2023-10-20T03:36:28.453 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 31.323333817000002, 2023-10-20T03:36:28.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019442797000000001, 2023-10-20T03:36:28.454 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.454 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:28.454 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:28.454 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:28.454 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:28.455 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 619 2023-10-20T03:36:28.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.455 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009377+0000", 2023-10-20T03:36:28.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009377+0000", 2023-10-20T03:36:28.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009381+0000", 2023-10-20T03:36:28.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2739999999999999e-06 2023-10-20T03:36:28.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009397+0000", 2023-10-20T03:36:28.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6549999999999999e-05 2023-10-20T03:36:28.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009442+0000", 2023-10-20T03:36:28.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4682999999999999e-05 2023-10-20T03:36:28.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009473+0000", 2023-10-20T03:36:28.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1163e-05 2023-10-20T03:36:28.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009642+0000", 2023-10-20T03:36:28.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016918200000000001 2023-10-20T03:36:28.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op must wait for map", 2023-10-20T03:36:28.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009668+0000", 2023-10-20T03:36:28.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6230000000000001e-05 2023-10-20T03:36:28.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.015833+0000", 2023-10-20T03:36:28.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0061643990000000001 2023-10-20T03:36:28.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:28.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.016114+0000", 2023-10-20T03:36:28.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028099199999999998 2023-10-20T03:36:28.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.028820+0000", 2023-10-20T03:36:28.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.012706324 2023-10-20T03:36:28.466 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.467 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.467 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.467 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.467 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:620 3.ds0 3:bbaca854:::smithi17039300-44:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2023-10-20T03:36:28.468 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:57.009528+0000", 2023-10-20T03:36:28.468 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 31.323183121, 2023-10-20T03:36:28.468 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021296131999999999, 2023-10-20T03:36:28.468 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.468 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:28.469 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:28.469 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:28.469 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:28.469 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 620 2023-10-20T03:36:28.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.470 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.470 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.470 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009528+0000", 2023-10-20T03:36:28.470 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.471 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.471 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009528+0000", 2023-10-20T03:36:28.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009530+0000", 2023-10-20T03:36:28.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.637e-06 2023-10-20T03:36:28.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.473 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.473 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009541+0000", 2023-10-20T03:36:28.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.133e-05 2023-10-20T03:36:28.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.474 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.474 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009564+0000", 2023-10-20T03:36:28.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3220000000000001e-05 2023-10-20T03:36:28.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.475 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009576+0000", 2023-10-20T03:36:28.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1419e-05 2023-10-20T03:36:28.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009879+0000", 2023-10-20T03:36:28.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030328100000000003 2023-10-20T03:36:28.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:28.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009902+0000", 2023-10-20T03:36:28.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3507e-05 2023-10-20T03:36:28.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.479 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.481 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.024765+0000", 2023-10-20T03:36:28.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014862548 2023-10-20T03:36:28.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.482 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:28.483 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.025046+0000", 2023-10-20T03:36:28.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028078100000000002 2023-10-20T03:36:28.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.483 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.484 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.030824+0000", 2023-10-20T03:36:28.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005778409 2023-10-20T03:36:28.484 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.484 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.484 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.485 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:621 3.ds0 3:bbaca854:::smithi17039300-44:head [read 0~737280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2023-10-20T03:36:28.485 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:57.009600+0000", 2023-10-20T03:36:28.485 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 31.323111411999999, 2023-10-20T03:36:28.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.02940736, 2023-10-20T03:36:28.486 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.486 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:28.486 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:28.486 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:28.486 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:28.487 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 621 2023-10-20T03:36:28.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.487 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.488 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.488 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009600+0000", 2023-10-20T03:36:28.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.488 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.489 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009600+0000", 2023-10-20T03:36:28.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.489 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.490 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009601+0000", 2023-10-20T03:36:28.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7700000000000003e-07 2023-10-20T03:36:28.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.490 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009606+0000", 2023-10-20T03:36:28.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6699999999999999e-06 2023-10-20T03:36:28.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.492 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009617+0000", 2023-10-20T03:36:28.492 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0643e-05 2023-10-20T03:36:28.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.493 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.009625+0000", 2023-10-20T03:36:28.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2130000000000008e-06 2023-10-20T03:36:28.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.494 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.010959+0000", 2023-10-20T03:36:28.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001334277 2023-10-20T03:36:28.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:28.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.010984+0000", 2023-10-20T03:36:28.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4487000000000001e-05 2023-10-20T03:36:28.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.496 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.496 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.030984+0000", 2023-10-20T03:36:28.496 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019999751999999999 2023-10-20T03:36:28.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.496 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:28.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.031305+0000", 2023-10-20T03:36:28.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032115399999999998 2023-10-20T03:36:28.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.498 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:57.039007+0000", 2023-10-20T03:36:28.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0077022870000000004 2023-10-20T03:36:28.498 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.498 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.498 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.499 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:653 3.ds0 3:beae359d:::smithi17039300-10:head [append 0~458752] snapc 2e=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2023-10-20T03:36:28.499 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:02.812132+0000", 2023-10-20T03:36:28.499 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.5205795, 2023-10-20T03:36:28.500 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.024528365, 2023-10-20T03:36:28.500 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.500 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:28.500 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:28.500 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:28.500 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:28.501 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 653 2023-10-20T03:36:28.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.501 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.812132+0000", 2023-10-20T03:36:28.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.812132+0000", 2023-10-20T03:36:28.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.812134+0000", 2023-10-20T03:36:28.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.844e-06 2023-10-20T03:36:28.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.812873+0000", 2023-10-20T03:36:28.505 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00073985499999999998 2023-10-20T03:36:28.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.505 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.505 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.812889+0000", 2023-10-20T03:36:28.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5597999999999999e-05 2023-10-20T03:36:28.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.506 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.506 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.506 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.812899+0000", 2023-10-20T03:36:28.507 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.6870000000000006e-06 2023-10-20T03:36:28.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.507 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.507 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.813000+0000", 2023-10-20T03:36:28.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000101135 2023-10-20T03:36:28.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.508 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:28.508 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.813181+0000", 2023-10-20T03:36:28.509 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000181628 2023-10-20T03:36:28.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.509 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:28.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.814450+0000", 2023-10-20T03:36:28.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001268307 2023-10-20T03:36:28.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:28.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.828274+0000", 2023-10-20T03:36:28.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013824336 2023-10-20T03:36:28.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:36:28.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.836513+0000", 2023-10-20T03:36:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0082384180000000008 2023-10-20T03:36:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.836660+0000", 2023-10-20T03:36:28.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014755700000000001 2023-10-20T03:36:28.513 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.513 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.513 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.514 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:654 3.ds0 3:beae359d:::smithi17039300-10:head [append 0~507904] snapc 2e=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2023-10-20T03:36:28.514 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:02.819491+0000", 2023-10-20T03:36:28.514 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.513220187000002, 2023-10-20T03:36:28.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027113301999999999, 2023-10-20T03:36:28.515 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.515 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:28.515 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:28.515 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:28.515 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:28.515 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 654 2023-10-20T03:36:28.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.516 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.819491+0000", 2023-10-20T03:36:28.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.517 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.819491+0000", 2023-10-20T03:36:28.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.518 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.819492+0000", 2023-10-20T03:36:28.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.333e-06 2023-10-20T03:36:28.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.820014+0000", 2023-10-20T03:36:28.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052147700000000001 2023-10-20T03:36:28.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.820032+0000", 2023-10-20T03:36:28.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8033000000000002e-05 2023-10-20T03:36:28.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.820041+0000", 2023-10-20T03:36:28.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7329999999999992e-06 2023-10-20T03:36:28.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.522 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.522 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.821894+0000", 2023-10-20T03:36:28.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0018534350000000001 2023-10-20T03:36:28.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:28.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.822039+0000", 2023-10-20T03:36:28.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000145191 2023-10-20T03:36:28.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:28.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.822900+0000", 2023-10-20T03:36:28.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000861124 2023-10-20T03:36:28.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:28.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.835974+0000", 2023-10-20T03:36:28.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013073939999999999 2023-10-20T03:36:28.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:36:28.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.846495+0000", 2023-10-20T03:36:28.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010520336999999999 2023-10-20T03:36:28.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.846604+0000", 2023-10-20T03:36:28.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010969899999999999 2023-10-20T03:36:28.527 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.528 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.528 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.528 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:655 3.ds0 3:beae359d:::smithi17039300-10:head [append 0~507904] snapc 2e=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2023-10-20T03:36:28.528 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:02.826594+0000", 2023-10-20T03:36:28.529 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.506117599, 2023-10-20T03:36:28.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020699608000000001, 2023-10-20T03:36:28.529 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.529 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:28.529 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:28.530 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:28.530 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:28.530 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 655 2023-10-20T03:36:28.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.530 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.826594+0000", 2023-10-20T03:36:28.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.826594+0000", 2023-10-20T03:36:28.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.826596+0000", 2023-10-20T03:36:28.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1550000000000001e-06 2023-10-20T03:36:28.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.534 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.534 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.827215+0000", 2023-10-20T03:36:28.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00061896300000000004 2023-10-20T03:36:28.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.827237+0000", 2023-10-20T03:36:28.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.207e-05 2023-10-20T03:36:28.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.827248+0000", 2023-10-20T03:36:28.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1649e-05 2023-10-20T03:36:28.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.828662+0000", 2023-10-20T03:36:28.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00141314 2023-10-20T03:36:28.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:28.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.828805+0000", 2023-10-20T03:36:28.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014317099999999999 2023-10-20T03:36:28.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:28.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.829931+0000", 2023-10-20T03:36:28.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0011262469999999999 2023-10-20T03:36:28.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:28.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.836852+0000", 2023-10-20T03:36:28.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0069211100000000003 2023-10-20T03:36:28.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:36:28.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.847184+0000", 2023-10-20T03:36:28.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010332328 2023-10-20T03:36:28.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:02.847293+0000", 2023-10-20T03:36:28.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000108775 2023-10-20T03:36:28.542 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.542 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.542 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.543 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:907 3.ds0 3:ba203010:::smithi17039300-4:head [copy-from ver 56] snapc 43=[43,42] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2023-10-20T03:36:28.543 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:24.253451+0000", 2023-10-20T03:36:28.543 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.079259822, 2023-10-20T03:36:28.543 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.093108890999999999, 2023-10-20T03:36:28.543 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.544 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2023-10-20T03:36:28.544 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:28.544 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:28.544 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:28.544 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 907 2023-10-20T03:36:28.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.545 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253451+0000", 2023-10-20T03:36:28.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253451+0000", 2023-10-20T03:36:28.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253454+0000", 2023-10-20T03:36:28.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6369999999999999e-06 2023-10-20T03:36:28.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.548 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.548 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253470+0000", 2023-10-20T03:36:28.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5645999999999999e-05 2023-10-20T03:36:28.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253503+0000", 2023-10-20T03:36:28.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3810000000000003e-05 2023-10-20T03:36:28.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253519+0000", 2023-10-20T03:36:28.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5823000000000001e-05 2023-10-20T03:36:28.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253650+0000", 2023-10-20T03:36:28.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013068500000000001 2023-10-20T03:36:28.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op must wait for map", 2023-10-20T03:36:28.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253661+0000", 2023-10-20T03:36:28.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1396999999999999e-05 2023-10-20T03:36:28.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.553 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.261146+0000", 2023-10-20T03:36:28.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0074843449999999999 2023-10-20T03:36:28.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:28.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.261240+0000", 2023-10-20T03:36:28.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3832000000000006e-05 2023-10-20T03:36:28.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:28.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.298838+0000", 2023-10-20T03:36:28.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.037598749000000001 2023-10-20T03:36:28.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:28.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.337825+0000", 2023-10-20T03:36:28.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038986357999999999 2023-10-20T03:36:28.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2023-10-20T03:36:28.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.346474+0000", 2023-10-20T03:36:28.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0086496509999999995 2023-10-20T03:36:28.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.346560+0000", 2023-10-20T03:36:28.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5958000000000006e-05 2023-10-20T03:36:28.563 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.564 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.564 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.565 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:908 3.ds0 3:ba203010:::smithi17039300-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e102)", 2023-10-20T03:36:28.566 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:24.253560+0000", 2023-10-20T03:36:28.566 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.0791508600000004, 2023-10-20T03:36:28.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.083230698000000006, 2023-10-20T03:36:28.567 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.567 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:28.567 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:28.568 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:28.568 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:28.569 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 908 2023-10-20T03:36:28.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.569 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.570 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.570 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253560+0000", 2023-10-20T03:36:28.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253560+0000", 2023-10-20T03:36:28.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.574 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253561+0000", 2023-10-20T03:36:28.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9999999999999996e-07 2023-10-20T03:36:28.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253567+0000", 2023-10-20T03:36:28.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0499999999999997e-06 2023-10-20T03:36:28.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.578 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253584+0000", 2023-10-20T03:36:28.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6497000000000001e-05 2023-10-20T03:36:28.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253593+0000", 2023-10-20T03:36:28.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3500000000000003e-06 2023-10-20T03:36:28.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253813+0000", 2023-10-20T03:36:28.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022003800000000001 2023-10-20T03:36:28.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.583 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting_for_map not empty", 2023-10-20T03:36:28.584 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.253829+0000", 2023-10-20T03:36:28.584 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5460000000000001e-05 2023-10-20T03:36:28.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.261404+0000", 2023-10-20T03:36:28.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0075754230000000004 2023-10-20T03:36:28.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for blocked object", 2023-10-20T03:36:28.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.261455+0000", 2023-10-20T03:36:28.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0677000000000003e-05 2023-10-20T03:36:28.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.336564+0000", 2023-10-20T03:36:28.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.075109123 2023-10-20T03:36:28.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:28.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.336666+0000", 2023-10-20T03:36:28.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010210899999999999 2023-10-20T03:36:28.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.336791+0000", 2023-10-20T03:36:28.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012507100000000001 2023-10-20T03:36:28.593 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.594 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.594 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.595 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4570.0:933 3.4s0 3:28b9edd9:::smithi17039300-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e102)", 2023-10-20T03:36:28.596 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:24.267981+0000", 2023-10-20T03:36:28.596 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.0647297760000001, 2023-10-20T03:36:28.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.025317474, 2023-10-20T03:36:28.597 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.597 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2023-10-20T03:36:28.597 INFO:teuthology.orchestra.run.smithi040.stdout: "client_info": { 2023-10-20T03:36:28.598 INFO:teuthology.orchestra.run.smithi040.stdout: "client": "client.4570", 2023-10-20T03:36:28.598 INFO:teuthology.orchestra.run.smithi040.stdout: "client_addr": "172.21.15.170:0/464918562", 2023-10-20T03:36:28.599 INFO:teuthology.orchestra.run.smithi040.stdout: "tid": 933 2023-10-20T03:36:28.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.599 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.267981+0000", 2023-10-20T03:36:28.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.267981+0000", 2023-10-20T03:36:28.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.267983+0000", 2023-10-20T03:36:28.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4929999999999999e-06 2023-10-20T03:36:28.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.267989+0000", 2023-10-20T03:36:28.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4799999999999998e-06 2023-10-20T03:36:28.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.268011+0000", 2023-10-20T03:36:28.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1452999999999999e-05 2023-10-20T03:36:28.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.268027+0000", 2023-10-20T03:36:28.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6336000000000001e-05 2023-10-20T03:36:28.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.274174+0000", 2023-10-20T03:36:28.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0061469799999999998 2023-10-20T03:36:28.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2023-10-20T03:36:28.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.274448+0000", 2023-10-20T03:36:28.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00027396099999999997 2023-10-20T03:36:28.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.616 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.287644+0000", 2023-10-20T03:36:28.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013195909 2023-10-20T03:36:28.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2023-10-20T03:36:28.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.287987+0000", 2023-10-20T03:36:28.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034285499999999998 2023-10-20T03:36:28.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.293299+0000", 2023-10-20T03:36:28.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053120069999999997 2023-10-20T03:36:28.620 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.621 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.621 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.622 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 102/56 ECSubWriteReply(tid=124, last_complete=102'90, committed=1, applied=1))", 2023-10-20T03:36:28.623 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:24.316766+0000", 2023-10-20T03:36:28.623 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.015945662, 2023-10-20T03:36:28.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020075148000000001, 2023-10-20T03:36:28.624 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.624 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:28.624 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.316766+0000", 2023-10-20T03:36:28.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.316766+0000", 2023-10-20T03:36:28.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.316767+0000", 2023-10-20T03:36:28.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.046e-06 2023-10-20T03:36:28.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.316772+0000", 2023-10-20T03:36:28.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1200000000000001e-06 2023-10-20T03:36:28.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.316786+0000", 2023-10-20T03:36:28.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4154e-05 2023-10-20T03:36:28.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.316794+0000", 2023-10-20T03:36:28.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4239999999999993e-06 2023-10-20T03:36:28.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.630 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.630 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.336827+0000", 2023-10-20T03:36:28.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020032439999999999 2023-10-20T03:36:28.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.336841+0000", 2023-10-20T03:36:28.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3964e-05 2023-10-20T03:36:28.632 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.632 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.632 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.633 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 102/56 ECSubWriteReply(tid=124, last_complete=102'90, committed=1, applied=1))", 2023-10-20T03:36:28.633 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:24.317677+0000", 2023-10-20T03:36:28.633 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.0150345099999996, 2023-10-20T03:36:28.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019188318999999999, 2023-10-20T03:36:28.633 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:28.633 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:28.634 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:28.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.634 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:28.634 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.317677+0000", 2023-10-20T03:36:28.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.635 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:28.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.317677+0000", 2023-10-20T03:36:28.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:28.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:28.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.317678+0000", 2023-10-20T03:36:28.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.66e-06 2023-10-20T03:36:28.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:28.637 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.317687+0000", 2023-10-20T03:36:28.637 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8300000000000002e-06 2023-10-20T03:36:28.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:28.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.317716+0000", 2023-10-20T03:36:28.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9067000000000001e-05 2023-10-20T03:36:28.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:28.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.317731+0000", 2023-10-20T03:36:28.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4269e-05 2023-10-20T03:36:28.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:28.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.336856+0000", 2023-10-20T03:36:28.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019125065 2023-10-20T03:36:28.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:28.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:28.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:28.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.336865+0000", 2023-10-20T03:36:28.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4280000000000002e-06 2023-10-20T03:36:28.641 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.642 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.642 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.642 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:28.642 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:28.642 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:28.645 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-10-20T03:36:28.646 INFO:tasks.rados.rados.0.smithi170.stdout:395: writing smithi17039300-19 from 3686400 to 3915776 tid 1 2023-10-20T03:36:28.646 INFO:tasks.rados.rados.0.smithi170.stdout:396: rollback oid 12 current snap is 35 2023-10-20T03:36:28.646 INFO:tasks.rados.rados.0.smithi170.stdout:396: done (1 left) 2023-10-20T03:36:28.646 INFO:tasks.rados.rados.0.smithi170.stdout:397: read oid 41 snap -1 2023-10-20T03:36:28.647 INFO:tasks.rados.rados.0.smithi170.stdout:397: expect (ObjNum 116 snap 22 seq_num 116) 2023-10-20T03:36:28.647 INFO:tasks.rados.rados.0.smithi170.stdout:398: read oid 4 snap -1 2023-10-20T03:36:28.647 INFO:tasks.rados.rados.0.smithi170.stdout:398: expect (ObjNum 150 snap 33 seq_num 150) 2023-10-20T03:36:28.647 INFO:tasks.rados.rados.0.smithi170.stdout:399: delete oid 15 current snap is 35 2023-10-20T03:36:28.647 INFO:tasks.rados.rados.0.smithi170.stdout:399: done (3 left) 2023-10-20T03:36:28.648 INFO:tasks.rados.rados.0.smithi170.stdout: waiting on 3 2023-10-20T03:36:28.648 INFO:tasks.rados.rados.0.smithi170.stdout:397: expect (ObjNum 116 snap 22 seq_num 116) 2023-10-20T03:36:28.648 INFO:tasks.rados.rados.0.smithi170.stdout:395: finishing write tid 1 to smithi17039300-19 2023-10-20T03:36:28.648 INFO:tasks.rados.rados.0.smithi170.stdout:395: finishing write tid 2 to smithi17039300-19 2023-10-20T03:36:28.648 INFO:tasks.rados.rados.0.smithi170.stdout:398: expect (ObjNum 150 snap 33 seq_num 150) 2023-10-20T03:36:28.649 INFO:tasks.rados.rados.0.smithi170.stdout:395: finishing write tid 3 to smithi17039300-19 2023-10-20T03:36:28.649 INFO:tasks.rados.rados.0.smithi170.stdout:395: oid 19 updating version 0 to 70 2023-10-20T03:36:28.649 INFO:tasks.rados.rados.0.smithi170.stdout:395: oid 19 version 70 is already newer than 69 2023-10-20T03:36:28.649 INFO:tasks.rados.rados.0.smithi170.stdout:update_object_version oid 19 v 70 (ObjNum 156 snap 35 seq_num 156) dirty exists 2023-10-20T03:36:28.649 INFO:tasks.rados.rados.0.smithi170.stdout:395: left oid 19 (ObjNum 156 snap 35 seq_num 156) 2023-10-20T03:36:28.650 INFO:tasks.rados.rados.0.smithi170.stdout:395: done (2 left) 2023-10-20T03:36:28.650 INFO:tasks.rados.rados.0.smithi170.stdout:397: done (1 left) 2023-10-20T03:36:28.650 INFO:tasks.rados.rados.0.smithi170.stdout:398: done (0 left) 2023-10-20T03:36:28.651 INFO:tasks.rados.rados.0.smithi170.stderr:0 errors. 2023-10-20T03:36:28.651 INFO:tasks.rados.rados.0.smithi170.stderr: 2023-10-20T03:36:28.651 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:36:28.652 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:36:28.643+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:36:28.745 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:28.743+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:28.786 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:28.786 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:36:28.787 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:36:28.787 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:28.801 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-10-20T03:36:28.846 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:28.844+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:28.928 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:28.928 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:36:28.928 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2023-10-20T03:36:28.928 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2023-10-20T03:36:28.929 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:28.942 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-10-20T03:36:28.947 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:36:28.940+0000 7f3068be6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:36:28.959 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:36:28.959 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":106,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:36:28.279734+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:35:35.398012+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.93946552276611328,"average_primary_affinity":0.93999999761581421,"average_primary_affinity_weighted":0.93999999761581421}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":0.93946552276611328,"average_primary_affinity":0.93999999761581421,"average_primary_affinity_weighted":0.93999999761581421}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-10-20T03:35:12.214840+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":71,"snap_epoch":106,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":0.072998046875,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":0.9584503173828125,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:24.064001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.884878+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.632626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:21.281410+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:03.397982+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:03.463084+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.320469+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:58.321221+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.321913+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:02.082859+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:03.991453+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":65536,"last_purged_snaps_scrub":"2023-10-20T03:36:05.214755+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:01.467282+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:25.210154+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:22.189560+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.6","mappings":[{"from":8,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":69,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:36:28.979 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T03:36:29.047 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:36:29.044+0000 7f23088c5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:36:29.123 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:29.123 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2023-10-20T03:36:29.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2023-10-20T03:36:29.123 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2023-10-20T03:36:29.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.124 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 44/31 ECSubWrite(tid=51, reqid=client.4570.0:333, at_version=44'18, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:29.124 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.387934+0000", 2023-10-20T03:36:29.124 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 62.731907333000002, 2023-10-20T03:36:29.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033593076999999999, 2023-10-20T03:36:29.125 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.125 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.125 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387934+0000", 2023-10-20T03:36:29.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387934+0000", 2023-10-20T03:36:29.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.387934+0000", 2023-10-20T03:36:29.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7e-07 2023-10-20T03:36:29.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388584+0000", 2023-10-20T03:36:29.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00064982500000000003 2023-10-20T03:36:29.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.130 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388724+0000", 2023-10-20T03:36:29.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013997200000000001 2023-10-20T03:36:29.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388764+0000", 2023-10-20T03:36:29.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0352999999999998e-05 2023-10-20T03:36:29.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388939+0000", 2023-10-20T03:36:29.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017499200000000001 2023-10-20T03:36:29.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.388956+0000", 2023-10-20T03:36:29.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6990000000000002e-05 2023-10-20T03:36:29.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.134 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421453+0000", 2023-10-20T03:36:29.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032496615999999999 2023-10-20T03:36:29.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.135 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.135 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421527+0000", 2023-10-20T03:36:29.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3959000000000001e-05 2023-10-20T03:36:29.135 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.135 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.135 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.136 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 44/31 ECSubWrite(tid=22, reqid=client.4570.0:340, at_version=44'12, trim_to=0'0, roll_forward_to=43'10))", 2023-10-20T03:36:29.136 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:36:29.136 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 62.720931512999996, 2023-10-20T03:36:29.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022077208000000001, 2023-10-20T03:36:29.137 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.137 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.137 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:36:29.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.139 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.139 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398909+0000", 2023-10-20T03:36:29.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.398910+0000", 2023-10-20T03:36:29.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6400000000000002e-07 2023-10-20T03:36:29.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399138+0000", 2023-10-20T03:36:29.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022810800000000001 2023-10-20T03:36:29.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399169+0000", 2023-10-20T03:36:29.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0593000000000003e-05 2023-10-20T03:36:29.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.143 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399190+0000", 2023-10-20T03:36:29.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.12e-05 2023-10-20T03:36:29.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399353+0000", 2023-10-20T03:36:29.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016285200000000001 2023-10-20T03:36:29.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.399378+0000", 2023-10-20T03:36:29.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5077000000000001e-05 2023-10-20T03:36:29.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420889+0000", 2023-10-20T03:36:29.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021511188000000001 2023-10-20T03:36:29.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.147 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.147 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420987+0000", 2023-10-20T03:36:29.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7626000000000003e-05 2023-10-20T03:36:29.147 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.147 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.148 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.148 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 44/31 ECSubWrite(tid=52, reqid=client.4570.0:334, at_version=44'19, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:29.148 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.401022+0000", 2023-10-20T03:36:29.148 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 62.718818478999999, 2023-10-20T03:36:29.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020571461999999999, 2023-10-20T03:36:29.149 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.149 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.149 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401022+0000", 2023-10-20T03:36:29.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401022+0000", 2023-10-20T03:36:29.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401024+0000", 2023-10-20T03:36:29.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.877e-06 2023-10-20T03:36:29.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401473+0000", 2023-10-20T03:36:29.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044877699999999998 2023-10-20T03:36:29.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401507+0000", 2023-10-20T03:36:29.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4209999999999999e-05 2023-10-20T03:36:29.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.154 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401543+0000", 2023-10-20T03:36:29.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6003000000000001e-05 2023-10-20T03:36:29.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401767+0000", 2023-10-20T03:36:29.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022351200000000001 2023-10-20T03:36:29.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.401793+0000", 2023-10-20T03:36:29.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6529e-05 2023-10-20T03:36:29.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421560+0000", 2023-10-20T03:36:29.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019766305000000001 2023-10-20T03:36:29.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.421594+0000", 2023-10-20T03:36:29.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4248999999999997e-05 2023-10-20T03:36:29.159 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.159 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.159 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.160 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 44/31 ECSubWrite(tid=53, reqid=client.4570.0:335, at_version=44'20, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:29.160 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.402960+0000", 2023-10-20T03:36:29.160 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 62.716880445999998, 2023-10-20T03:36:29.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019889799999999999, 2023-10-20T03:36:29.160 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.161 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.161 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402960+0000", 2023-10-20T03:36:29.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402960+0000", 2023-10-20T03:36:29.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.402962+0000", 2023-10-20T03:36:29.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8470000000000001e-06 2023-10-20T03:36:29.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403065+0000", 2023-10-20T03:36:29.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000102179 2023-10-20T03:36:29.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403099+0000", 2023-10-20T03:36:29.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4813e-05 2023-10-20T03:36:29.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.403122+0000", 2023-10-20T03:36:29.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2226e-05 2023-10-20T03:36:29.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419074+0000", 2023-10-20T03:36:29.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015952643999999998 2023-10-20T03:36:29.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.419093+0000", 2023-10-20T03:36:29.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9089e-05 2023-10-20T03:36:29.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.422750+0000", 2023-10-20T03:36:29.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0036562959999999999 2023-10-20T03:36:29.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.170 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.422850+0000", 2023-10-20T03:36:29.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010070599999999999 2023-10-20T03:36:29.171 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.171 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.171 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.172 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 44/31 ECSubWrite(tid=54, reqid=client.4570.0:336, at_version=44'21, trim_to=0'0, roll_forward_to=39'16))", 2023-10-20T03:36:29.172 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:26.404639+0000", 2023-10-20T03:36:29.172 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 62.715201845999999, 2023-10-20T03:36:29.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.018626967000000001, 2023-10-20T03:36:29.173 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.173 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.173 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404639+0000", 2023-10-20T03:36:29.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.174 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.174 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404639+0000", 2023-10-20T03:36:29.175 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.175 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.175 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.175 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.175 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404641+0000", 2023-10-20T03:36:29.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7969999999999999e-06 2023-10-20T03:36:29.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.176 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.176 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404707+0000", 2023-10-20T03:36:29.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6523e-05 2023-10-20T03:36:29.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.177 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.177 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.177 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404741+0000", 2023-10-20T03:36:29.177 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4078999999999997e-05 2023-10-20T03:36:29.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.178 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.178 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.178 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.404764+0000", 2023-10-20T03:36:29.178 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.209e-05 2023-10-20T03:36:29.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.179 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.179 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420947+0000", 2023-10-20T03:36:29.179 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016183831999999999 2023-10-20T03:36:29.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.420971+0000", 2023-10-20T03:36:29.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.332e-05 2023-10-20T03:36:29.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.181 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.181 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.423172+0000", 2023-10-20T03:36:29.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002200867 2023-10-20T03:36:29.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:26.423266+0000", 2023-10-20T03:36:29.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4458999999999998e-05 2023-10-20T03:36:29.183 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.183 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.183 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.184 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 48/31 ECSubWrite(tid=60, reqid=client.4570.0:384, at_version=48'23, trim_to=0'0, roll_forward_to=44'21))", 2023-10-20T03:36:29.184 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:30.392373+0000", 2023-10-20T03:36:29.184 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 58.727467634999996, 2023-10-20T03:36:29.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.038515463, 2023-10-20T03:36:29.184 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.185 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.185 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.185 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.185 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.392373+0000", 2023-10-20T03:36:29.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.186 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.186 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.392373+0000", 2023-10-20T03:36:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.392374+0000", 2023-10-20T03:36:29.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3e-07 2023-10-20T03:36:29.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394109+0000", 2023-10-20T03:36:29.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001734977 2023-10-20T03:36:29.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394151+0000", 2023-10-20T03:36:29.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1872999999999997e-05 2023-10-20T03:36:29.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.190 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.190 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394177+0000", 2023-10-20T03:36:29.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5636e-05 2023-10-20T03:36:29.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.191 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394320+0000", 2023-10-20T03:36:29.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014345499999999999 2023-10-20T03:36:29.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.394348+0000", 2023-10-20T03:36:29.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7523999999999999e-05 2023-10-20T03:36:29.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.430793+0000", 2023-10-20T03:36:29.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.036445549000000001 2023-10-20T03:36:29.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.194 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:30.430889+0000", 2023-10-20T03:36:29.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5718999999999996e-05 2023-10-20T03:36:29.195 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.195 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.195 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.196 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 71/31 ECSubWrite(tid=52, reqid=client.4570.0:589, at_version=71'24, trim_to=0'0, roll_forward_to=61'23))", 2023-10-20T03:36:29.196 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.004084+0000", 2023-10-20T03:36:29.196 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 36.115757330999998, 2023-10-20T03:36:29.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.041882438000000001, 2023-10-20T03:36:29.196 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.197 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.197 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.004084+0000", 2023-10-20T03:36:29.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.198 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.198 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.004084+0000", 2023-10-20T03:36:29.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.004086+0000", 2023-10-20T03:36:29.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8499999999999998e-06 2023-10-20T03:36:29.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.005054+0000", 2023-10-20T03:36:29.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00096745999999999996 2023-10-20T03:36:29.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.201 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.201 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.005115+0000", 2023-10-20T03:36:29.201 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0888999999999999e-05 2023-10-20T03:36:29.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.005144+0000", 2023-10-20T03:36:29.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9122999999999999e-05 2023-10-20T03:36:29.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.203 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.203 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.005300+0000", 2023-10-20T03:36:29.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000156269 2023-10-20T03:36:29.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.204 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.204 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.005316+0000", 2023-10-20T03:36:29.204 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6277e-05 2023-10-20T03:36:29.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.045929+0000", 2023-10-20T03:36:29.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.040612662000000001 2023-10-20T03:36:29.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.206 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.206 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.045966+0000", 2023-10-20T03:36:29.206 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6908e-05 2023-10-20T03:36:29.206 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.207 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.207 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.207 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.207 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 71/31 ECSubWrite(tid=53, reqid=client.4570.0:590, at_version=71'25, trim_to=0'0, roll_forward_to=61'23))", 2023-10-20T03:36:29.207 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:35:53.013561+0000", 2023-10-20T03:36:29.208 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 36.106280392000002, 2023-10-20T03:36:29.208 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.032941350000000001, 2023-10-20T03:36:29.208 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.208 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.208 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.209 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.209 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.013561+0000", 2023-10-20T03:36:29.209 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.210 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.013561+0000", 2023-10-20T03:36:29.210 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.211 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.211 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.013563+0000", 2023-10-20T03:36:29.211 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5600000000000001e-06 2023-10-20T03:36:29.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.212 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.212 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.014144+0000", 2023-10-20T03:36:29.212 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00058107500000000002 2023-10-20T03:36:29.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.213 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.213 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.014197+0000", 2023-10-20T03:36:29.213 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3257000000000002e-05 2023-10-20T03:36:29.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.214 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.014223+0000", 2023-10-20T03:36:29.214 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6006000000000001e-05 2023-10-20T03:36:29.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.024055+0000", 2023-10-20T03:36:29.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0098312320000000005 2023-10-20T03:36:29.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.216 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.216 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.024090+0000", 2023-10-20T03:36:29.216 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5309999999999999e-05 2023-10-20T03:36:29.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.217 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.217 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.046390+0000", 2023-10-20T03:36:29.217 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.022299864999999999 2023-10-20T03:36:29.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.218 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.218 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:35:53.046502+0000", 2023-10-20T03:36:29.218 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112045 2023-10-20T03:36:29.218 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.218 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.219 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.219 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 86/31 ECSubWrite(tid=73, reqid=client.4570.0:713, at_version=86'38, trim_to=0'0, roll_forward_to=85'36))", 2023-10-20T03:36:29.219 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:08.121970+0000", 2023-10-20T03:36:29.220 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.997870536000001, 2023-10-20T03:36:29.220 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.031323551999999998, 2023-10-20T03:36:29.220 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.220 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.220 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.221 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.221 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.121970+0000", 2023-10-20T03:36:29.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.222 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.222 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.121970+0000", 2023-10-20T03:36:29.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.121971+0000", 2023-10-20T03:36:29.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.77e-07 2023-10-20T03:36:29.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.122439+0000", 2023-10-20T03:36:29.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046788 2023-10-20T03:36:29.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.122465+0000", 2023-10-20T03:36:29.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6570000000000001e-05 2023-10-20T03:36:29.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.122480+0000", 2023-10-20T03:36:29.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5153e-05 2023-10-20T03:36:29.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.227 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.122655+0000", 2023-10-20T03:36:29.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017485100000000001 2023-10-20T03:36:29.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.122669+0000", 2023-10-20T03:36:29.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.42e-05 2023-10-20T03:36:29.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.228 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.153179+0000", 2023-10-20T03:36:29.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.030509535000000001 2023-10-20T03:36:29.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.153294+0000", 2023-10-20T03:36:29.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000114986 2023-10-20T03:36:29.230 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.230 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.230 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.231 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 86/31 ECSubWrite(tid=74, reqid=client.4570.0:714, at_version=86'39, trim_to=0'0, roll_forward_to=85'36))", 2023-10-20T03:36:29.231 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:08.134886+0000", 2023-10-20T03:36:29.231 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.984954869999999, 2023-10-20T03:36:29.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.033027405000000003, 2023-10-20T03:36:29.232 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.232 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.232 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.134886+0000", 2023-10-20T03:36:29.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.233 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.134886+0000", 2023-10-20T03:36:29.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.134888+0000", 2023-10-20T03:36:29.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5230000000000001e-06 2023-10-20T03:36:29.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.135317+0000", 2023-10-20T03:36:29.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00042957999999999999 2023-10-20T03:36:29.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.135349+0000", 2023-10-20T03:36:29.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1640000000000002e-05 2023-10-20T03:36:29.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.237 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.237 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.135366+0000", 2023-10-20T03:36:29.238 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6869999999999999e-05 2023-10-20T03:36:29.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.238 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.238 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.139875+0000", 2023-10-20T03:36:29.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045092209999999999 2023-10-20T03:36:29.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.139905+0000", 2023-10-20T03:36:29.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0583999999999997e-05 2023-10-20T03:36:29.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.167809+0000", 2023-10-20T03:36:29.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.027903568 2023-10-20T03:36:29.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.167913+0000", 2023-10-20T03:36:29.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104419 2023-10-20T03:36:29.242 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.242 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.242 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.243 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 86/31 ECSubWrite(tid=75, reqid=client.4570.0:715, at_version=86'40, trim_to=0'0, roll_forward_to=85'36))", 2023-10-20T03:36:29.243 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:08.148814+0000", 2023-10-20T03:36:29.243 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.971027007, 2023-10-20T03:36:29.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.019154707999999999, 2023-10-20T03:36:29.244 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.244 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.244 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.148814+0000", 2023-10-20T03:36:29.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.148814+0000", 2023-10-20T03:36:29.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.247 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.148816+0000", 2023-10-20T03:36:29.247 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.074e-06 2023-10-20T03:36:29.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.247 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.248 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.149031+0000", 2023-10-20T03:36:29.248 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021531799999999999 2023-10-20T03:36:29.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.248 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.248 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.249 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.149063+0000", 2023-10-20T03:36:29.249 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1702999999999998e-05 2023-10-20T03:36:29.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.149088+0000", 2023-10-20T03:36:29.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4655999999999999e-05 2023-10-20T03:36:29.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.153391+0000", 2023-10-20T03:36:29.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043032950000000004 2023-10-20T03:36:29.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.251 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.153409+0000", 2023-10-20T03:36:29.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7589999999999999e-05 2023-10-20T03:36:29.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.252 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.167925+0000", 2023-10-20T03:36:29.253 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014516416000000001 2023-10-20T03:36:29.253 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.253 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.253 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.253 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:08.167969+0000", 2023-10-20T03:36:29.254 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3655999999999997e-05 2023-10-20T03:36:29.254 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.254 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.254 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.255 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 92/56 ECSubWrite(tid=100, reqid=client.4570.0:799, at_version=92'71, trim_to=0'0, roll_forward_to=92'69))", 2023-10-20T03:36:29.255 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:14.369327+0000", 2023-10-20T03:36:29.255 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 14.750513508999999, 2023-10-20T03:36:29.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016897387, 2023-10-20T03:36:29.256 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.256 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.256 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.369327+0000", 2023-10-20T03:36:29.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.257 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.369327+0000", 2023-10-20T03:36:29.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.258 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.369330+0000", 2023-10-20T03:36:29.259 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.193e-06 2023-10-20T03:36:29.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.259 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.259 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.259 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.369595+0000", 2023-10-20T03:36:29.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000265641 2023-10-20T03:36:29.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.369630+0000", 2023-10-20T03:36:29.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4306e-05 2023-10-20T03:36:29.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.369651+0000", 2023-10-20T03:36:29.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1226999999999999e-05 2023-10-20T03:36:29.262 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.262 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.262 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.262 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.369864+0000", 2023-10-20T03:36:29.262 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021313500000000001 2023-10-20T03:36:29.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.263 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.263 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.369905+0000", 2023-10-20T03:36:29.263 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1009e-05 2023-10-20T03:36:29.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.264 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.264 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.264 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.386109+0000", 2023-10-20T03:36:29.264 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016203584 2023-10-20T03:36:29.264 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.265 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.265 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.265 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.386225+0000", 2023-10-20T03:36:29.265 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000116292 2023-10-20T03:36:29.265 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.265 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.266 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.266 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.266 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.266 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 92/56 ECSubWrite(tid=101, reqid=client.4570.0:800, at_version=92'72, trim_to=0'0, roll_forward_to=92'69))", 2023-10-20T03:36:29.266 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:14.388150+0000", 2023-10-20T03:36:29.267 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 14.731690686, 2023-10-20T03:36:29.267 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.016211079999999999, 2023-10-20T03:36:29.267 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.267 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.267 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.388150+0000", 2023-10-20T03:36:29.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.269 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.269 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.388150+0000", 2023-10-20T03:36:29.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.270 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.270 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.388153+0000", 2023-10-20T03:36:29.270 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5730000000000002e-06 2023-10-20T03:36:29.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.270 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.271 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.271 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.388555+0000", 2023-10-20T03:36:29.271 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040195699999999999 2023-10-20T03:36:29.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.271 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.272 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.272 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.388596+0000", 2023-10-20T03:36:29.272 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1183000000000002e-05 2023-10-20T03:36:29.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.272 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.272 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.273 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.388622+0000", 2023-10-20T03:36:29.273 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6203e-05 2023-10-20T03:36:29.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.273 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.273 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.274 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.388850+0000", 2023-10-20T03:36:29.274 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000227975 2023-10-20T03:36:29.274 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.274 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.274 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.275 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.388890+0000", 2023-10-20T03:36:29.275 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0197000000000002e-05 2023-10-20T03:36:29.275 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.275 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.275 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.276 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.404246+0000", 2023-10-20T03:36:29.276 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015356053 2023-10-20T03:36:29.276 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.276 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.276 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.277 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.404361+0000", 2023-10-20T03:36:29.277 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011493900000000001 2023-10-20T03:36:29.277 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.277 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.277 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.278 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.278 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.278 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 92/56 ECSubWrite(tid=102, reqid=client.4570.0:801, at_version=92'73, trim_to=0'0, roll_forward_to=92'69))", 2023-10-20T03:36:29.278 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:14.405636+0000", 2023-10-20T03:36:29.278 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 14.714204459999999, 2023-10-20T03:36:29.279 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.021587503000000001, 2023-10-20T03:36:29.279 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.279 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.279 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.279 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.280 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.280 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.405636+0000", 2023-10-20T03:36:29.280 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.280 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.280 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.281 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.405636+0000", 2023-10-20T03:36:29.281 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.281 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.405639+0000", 2023-10-20T03:36:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.26e-06 2023-10-20T03:36:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.282 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.405945+0000", 2023-10-20T03:36:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000305982 2023-10-20T03:36:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.283 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.405985+0000", 2023-10-20T03:36:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0459e-05 2023-10-20T03:36:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.406009+0000", 2023-10-20T03:36:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3876e-05 2023-10-20T03:36:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.406178+0000", 2023-10-20T03:36:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000169159 2023-10-20T03:36:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.406220+0000", 2023-10-20T03:36:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1557000000000002e-05 2023-10-20T03:36:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.427129+0000", 2023-10-20T03:36:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020908998000000002 2023-10-20T03:36:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.288 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:14.427224+0000", 2023-10-20T03:36:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5211999999999996e-05 2023-10-20T03:36:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.289 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 97/92 ECSubWrite(tid=131, reqid=osd.11.0:131, at_version=97'70, trim_to=0'0, roll_forward_to=96'68))", 2023-10-20T03:36:29.290 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:19.191314+0000", 2023-10-20T03:36:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 9.9285266940000003, 2023-10-20T03:36:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020966887, 2023-10-20T03:36:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.191314+0000", 2023-10-20T03:36:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.191314+0000", 2023-10-20T03:36:29.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.191315+0000", 2023-10-20T03:36:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.3000000000000001e-07 2023-10-20T03:36:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.191382+0000", 2023-10-20T03:36:29.295 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7665999999999999e-05 2023-10-20T03:36:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.191418+0000", 2023-10-20T03:36:29.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5556999999999999e-05 2023-10-20T03:36:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.191448+0000", 2023-10-20T03:36:29.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9560000000000002e-05 2023-10-20T03:36:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.191628+0000", 2023-10-20T03:36:29.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018084500000000001 2023-10-20T03:36:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.191663+0000", 2023-10-20T03:36:29.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4663000000000003e-05 2023-10-20T03:36:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.212167+0000", 2023-10-20T03:36:29.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020503864 2023-10-20T03:36:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.212281+0000", 2023-10-20T03:36:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000114202 2023-10-20T03:36:29.304 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.305 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 97/92 ECSubWrite(tid=132, reqid=client.4570.0:841, at_version=97'71, trim_to=0'0, roll_forward_to=96'68))", 2023-10-20T03:36:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:19.193492+0000", 2023-10-20T03:36:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 9.9263488340000006, 2023-10-20T03:36:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01886987, 2023-10-20T03:36:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.306 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.193492+0000", 2023-10-20T03:36:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.193492+0000", 2023-10-20T03:36:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.193494+0000", 2023-10-20T03:36:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0339999999999999e-06 2023-10-20T03:36:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.193537+0000", 2023-10-20T03:36:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2633000000000003e-05 2023-10-20T03:36:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.193569+0000", 2023-10-20T03:36:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2400000000000001e-05 2023-10-20T03:36:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.193592+0000", 2023-10-20T03:36:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2543000000000001e-05 2023-10-20T03:36:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.312 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.194438+0000", 2023-10-20T03:36:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00084597999999999997 2023-10-20T03:36:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.194473+0000", 2023-10-20T03:36:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4900000000000001e-05 2023-10-20T03:36:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.212294+0000", 2023-10-20T03:36:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017820936999999999 2023-10-20T03:36:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.212362+0000", 2023-10-20T03:36:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8442999999999995e-05 2023-10-20T03:36:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 97/92 ECSubWrite(tid=133, reqid=client.4570.0:842, at_version=97'72, trim_to=0'0, roll_forward_to=96'68))", 2023-10-20T03:36:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:19.194560+0000", 2023-10-20T03:36:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 9.9252812880000008, 2023-10-20T03:36:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020940146999999999, 2023-10-20T03:36:29.317 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.194560+0000", 2023-10-20T03:36:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.194560+0000", 2023-10-20T03:36:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.194562+0000", 2023-10-20T03:36:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.886e-06 2023-10-20T03:36:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.194613+0000", 2023-10-20T03:36:29.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1943000000000001e-05 2023-10-20T03:36:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.194648+0000", 2023-10-20T03:36:29.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4844e-05 2023-10-20T03:36:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.194676+0000", 2023-10-20T03:36:29.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7793e-05 2023-10-20T03:36:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.195240+0000", 2023-10-20T03:36:29.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00056361200000000001 2023-10-20T03:36:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.195275+0000", 2023-10-20T03:36:29.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5660000000000001e-05 2023-10-20T03:36:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.215393+0000", 2023-10-20T03:36:29.326 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.020117172999999999 2023-10-20T03:36:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.215500+0000", 2023-10-20T03:36:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107236 2023-10-20T03:36:29.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.es6 97/92 ECSubWrite(tid=134, reqid=client.4570.0:843, at_version=97'73, trim_to=0'0, roll_forward_to=96'68))", 2023-10-20T03:36:29.328 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:19.198087+0000", 2023-10-20T03:36:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 9.9217537609999997, 2023-10-20T03:36:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017492429, 2023-10-20T03:36:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.329 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.198087+0000", 2023-10-20T03:36:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.198087+0000", 2023-10-20T03:36:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.198089+0000", 2023-10-20T03:36:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.933e-06 2023-10-20T03:36:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.198144+0000", 2023-10-20T03:36:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4639000000000003e-05 2023-10-20T03:36:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.198182+0000", 2023-10-20T03:36:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8096999999999998e-05 2023-10-20T03:36:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.198204+0000", 2023-10-20T03:36:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1840000000000001e-05 2023-10-20T03:36:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.207355+0000", 2023-10-20T03:36:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0091516329999999993 2023-10-20T03:36:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.336 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.207390+0000", 2023-10-20T03:36:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4529000000000001e-05 2023-10-20T03:36:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.215511+0000", 2023-10-20T03:36:29.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0081206990000000003 2023-10-20T03:36:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.215580+0000", 2023-10-20T03:36:29.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9059000000000004e-05 2023-10-20T03:36:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.339 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.5s2 97/94 ECSubWrite(tid=41, reqid=client.4570.0:844, at_version=97'32, trim_to=0'0, roll_forward_to=0'0))", 2023-10-20T03:36:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:19.199158+0000", 2023-10-20T03:36:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 9.92068248, 2023-10-20T03:36:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017779682000000002, 2023-10-20T03:36:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.340 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.199158+0000", 2023-10-20T03:36:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.199158+0000", 2023-10-20T03:36:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.199159+0000", 2023-10-20T03:36:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2023-10-20T03:36:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.343 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.199228+0000", 2023-10-20T03:36:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9070000000000001e-05 2023-10-20T03:36:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.199261+0000", 2023-10-20T03:36:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2923000000000003e-05 2023-10-20T03:36:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.199284+0000", 2023-10-20T03:36:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3e-05 2023-10-20T03:36:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.199562+0000", 2023-10-20T03:36:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00027801099999999999 2023-10-20T03:36:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.199596+0000", 2023-10-20T03:36:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3485999999999998e-05 2023-10-20T03:36:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.216851+0000", 2023-10-20T03:36:29.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017254716 2023-10-20T03:36:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:19.216938+0000", 2023-10-20T03:36:29.349 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7635999999999999e-05 2023-10-20T03:36:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "MOSDECSubOpWrite(3.ds6 102/56 ECSubWrite(tid=124, reqid=client.4570.0:907, at_version=102'90, trim_to=0'0, roll_forward_to=100'89))", 2023-10-20T03:36:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2023-10-20T03:36:24.299124+0000", 2023-10-20T03:36:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.8207169050000003, 2023-10-20T03:36:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.017518155000000001, 2023-10-20T03:36:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2023-10-20T03:36:29.351 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "reached pg", 2023-10-20T03:36:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2023-10-20T03:36:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2023-10-20T03:36:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.299124+0000", 2023-10-20T03:36:29.352 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2023-10-20T03:36:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.299124+0000", 2023-10-20T03:36:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2023-10-20T03:36:29.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2023-10-20T03:36:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.299125+0000", 2023-10-20T03:36:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9699999999999995e-07 2023-10-20T03:36:29.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2023-10-20T03:36:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.300460+0000", 2023-10-20T03:36:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00133494 2023-10-20T03:36:29.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2023-10-20T03:36:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.300515+0000", 2023-10-20T03:36:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5340000000000002e-05 2023-10-20T03:36:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2023-10-20T03:36:29.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.300538+0000", 2023-10-20T03:36:29.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3428e-05 2023-10-20T03:36:29.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2023-10-20T03:36:29.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.300648+0000", 2023-10-20T03:36:29.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109894 2023-10-20T03:36:29.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_started", 2023-10-20T03:36:29.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.300656+0000", 2023-10-20T03:36:29.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2010000000000002e-06 2023-10-20T03:36:29.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_committed", 2023-10-20T03:36:29.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.316603+0000", 2023-10-20T03:36:29.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015947459000000001 2023-10-20T03:36:29.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2023-10-20T03:36:29.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-10-20T03:36:29.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2023-10-20T03:36:29.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2023-10-20T03:36:24.316642+0000", 2023-10-20T03:36:29.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9196000000000002e-05 2023-10-20T03:36:29.361 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.361 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.361 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.361 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-10-20T03:36:29.361 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-10-20T03:36:29.362 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:29.362 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-10-20T03:36:29.365 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:36:29.366 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-10-20T03:36:29.367 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-10-20T03:36:29.368 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2023-10-20T03:36:28.730721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87671328,"num_objects":57,"num_object_clones":17,"num_object_copies":389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":2,"num_read":556,"num_read_kb":258262,"num_write":1147,"num_write_kb":141444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":91504640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1195,"ondisk_log_size":1195,"up":127,"acting":127,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":643324,"kb_used_data":167272,"kb_used_omap":28,"kb_used_meta":475811,"kb_avail":1414934276,"statfs":{"total":1449551462400,"available":1448892698624,"internally_reserved":0,"allocated":171286528,"data_stored":208151647,"data_compressed":2395564,"data_compressed_allocated":4952064,"data_compressed_original":10866688,"omap_allocated":29064,"internal_metadata":487231096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16465920,"num_objects":-7,"num_object_clones":-8,"num_object_copies":-49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7,"num_whiteouts":-4,"num_read":35,"num_read_kb":37293,"num_write":64,"num_write_kb":5232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001471"},"pg_stats":[{"pgid":"3.f","version":"103'96","reported_seq":394,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-20T03:36:25.275676+0000","last_change":"2023-10-20T03:36:25.277429+0000","last_active":"2023-10-20T03:36:25.275676+0000","last_peered":"2023-10-20T03:36:25.275676+0000","last_clean":"2023-10-20T03:36:25.275676+0000","last_became_active":"2023-10-20T03:35:30.382595+0000","last_became_peered":"2023-10-20T03:35:30.382595+0000","last_unstale":"2023-10-20T03:36:25.275676+0000","last_undegraded":"2023-10-20T03:36:25.275676+0000","last_fullsized":"2023-10-20T03:36:25.275676+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'78","last_scrub_stamp":"2023-10-20T03:36:19.084653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:36:19.084653+0000","objects_scrubbed":4,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:37:29.305616+0000","scrub_duration":0.012971172,"objects_trimmed":0,"snaptrim_duration":0.009944863,"stat_sum":{"num_bytes":4784128,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":8,"num_write":90,"num_write_kb":7408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,15,14,2,11,1],"acting":[0,3,15,14,2,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"4"},{"start":"3b","length":"1"},{"start":"3f","length":"1"},{"start":"42","length":"1"}]},{"pgid":"3.e","version":"100'82","reported_seq":430,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-20T03:36:25.271827+0000","last_change":"2023-10-20T03:36:25.273088+0000","last_active":"2023-10-20T03:36:25.271827+0000","last_peered":"2023-10-20T03:36:25.271827+0000","last_clean":"2023-10-20T03:36:25.271827+0000","last_became_active":"2023-10-20T03:36:15.161134+0000","last_became_peered":"2023-10-20T03:36:15.161134+0000","last_unstale":"2023-10-20T03:36:25.271827+0000","last_undegraded":"2023-10-20T03:36:25.271827+0000","last_fullsized":"2023-10-20T03:36:25.271827+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:33.809716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001445153,"stat_sum":{"num_bytes":6569984,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":17754,"num_write":107,"num_write_kb":14144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":14647296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,6,4,13,3,5],"acting":[11,14,6,4,13,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"4"},{"start":"3b","length":"1"},{"start":"3f","length":"1"},{"start":"42","length":"1"}]},{"pgid":"3.d","version":"102'90","reported_seq":390,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-20T03:36:25.277365+0000","last_change":"2023-10-20T03:36:25.278875+0000","last_active":"2023-10-20T03:36:25.277365+0000","last_peered":"2023-10-20T03:36:25.277365+0000","last_clean":"2023-10-20T03:36:25.277365+0000","last_became_active":"2023-10-20T03:35:38.820900+0000","last_became_peered":"2023-10-20T03:35:38.820900+0000","last_unstale":"2023-10-20T03:36:25.277365+0000","last_undegraded":"2023-10-20T03:36:25.277365+0000","last_fullsized":"2023-10-20T03:36:25.277365+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:41.150216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023079250000000002,"stat_sum":{"num_bytes":8830976,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":29247,"num_write":58,"num_write_kb":15440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,11,0,8,13,5],"acting":[1,4,11,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"4"},{"start":"3b","length":"1"},{"start":"3f","length":"1"},{"start":"42","length":"1"}]},{"pgid":"3.c","version":"102'72","reported_seq":365,"reported_epoch":103,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:36:25.274346+0000","last_change":"2023-10-20T03:36:17.195721+0000","last_active":"2023-10-20T03:36:25.274346+0000","last_peered":"2023-10-20T03:36:25.274346+0000","last_clean":"2023-10-20T03:36:15.184005+0000","last_became_active":"2023-10-20T03:36:17.195721+0000","last_became_peered":"2023-10-20T03:36:17.195721+0000","last_unstale":"2023-10-20T03:36:25.274346+0000","last_undegraded":"2023-10-20T03:36:17.177610+0000","last_fullsized":"2023-10-20T03:36:17.176354+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012586561,"stat_sum":{"num_bytes":2752512,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":17433,"num_write":29,"num_write_kb":5856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":573440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,5,3,7,1,2147483647],"acting":[0,13,5,3,7,1,2147483647],"avail_no_missing":["13(1)","0(0)","1(5)","3(3)","5(2)","7(4)"],"object_location_counts":[{"shards":"0(0),1(5),3(3),5(2),7(4),13(1)","objects":3}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"1"}]},{"pgid":"3.b","version":"103'58","reported_seq":375,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-20T03:36:25.293761+0000","last_change":"2023-10-20T03:36:25.296670+0000","last_active":"2023-10-20T03:36:25.293761+0000","last_peered":"2023-10-20T03:36:25.293761+0000","last_clean":"2023-10-20T03:36:25.293761+0000","last_became_active":"2023-10-20T03:35:38.816060+0000","last_became_peered":"2023-10-20T03:35:38.816060+0000","last_unstale":"2023-10-20T03:36:25.293761+0000","last_undegraded":"2023-10-20T03:36:25.293761+0000","last_fullsized":"2023-10-20T03:36:25.293761+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:26.302133+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.022515390999999999,"stat_sum":{"num_bytes":9617408,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":7389,"num_write":31,"num_write_kb":9792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2752512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,10,0,11,4,14],"acting":[13,1,10,0,11,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"4"},{"start":"3b","length":"1"},{"start":"3f","length":"1"},{"start":"42","length":"1"}]},{"pgid":"3.a","version":"102'68","reported_seq":279,"reported_epoch":103,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:36:25.396045+0000","last_change":"2023-10-20T03:36:17.187291+0000","last_active":"2023-10-20T03:36:25.396045+0000","last_peered":"2023-10-20T03:36:25.396045+0000","last_clean":"2023-10-20T03:36:15.260438+0000","last_became_active":"2023-10-20T03:36:17.187291+0000","last_became_peered":"2023-10-20T03:36:17.187291+0000","last_unstale":"2023-10-20T03:36:25.396045+0000","last_undegraded":"2023-10-20T03:36:17.170831+0000","last_fullsized":"2023-10-20T03:36:17.170510+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":13254656,"num_objects":14,"num_object_clones":10,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":32,"num_read_kb":25713,"num_write":147,"num_write_kb":13248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":10813440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,4,6,10,3,15,8],"acting":[2147483647,4,6,10,3,15,8],"avail_no_missing":["4(1)","3(4)","6(2)","8(6)","10(3)","15(5)"],"object_location_counts":[{"shards":"3(4),4(1),6(2),8(6),10(3),15(5)","objects":14}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"101'56","reported_seq":329,"reported_epoch":103,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:36:25.283760+0000","last_change":"2023-10-20T03:36:17.186951+0000","last_active":"2023-10-20T03:36:25.283760+0000","last_peered":"2023-10-20T03:36:25.283760+0000","last_clean":"2023-10-20T03:36:15.143467+0000","last_became_active":"2023-10-20T03:36:17.186951+0000","last_became_peered":"2023-10-20T03:36:17.186951+0000","last_unstale":"2023-10-20T03:36:25.283760+0000","last_undegraded":"2023-10-20T03:36:17.175240+0000","last_fullsized":"2023-10-20T03:36:17.174150+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010593429999999999,"stat_sum":{"num_bytes":5423104,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":6490,"num_write":30,"num_write_kb":9696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,10,11,4,2147483647,3],"acting":[7,2,10,11,4,2147483647,3],"avail_no_missing":["7(0)","2(1)","3(6)","4(4)","10(2)","11(3)"],"object_location_counts":[{"shards":"2(1),3(6),4(4),7(0),10(2),11(3)","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"1"}]},{"pgid":"3.8","version":"100'35","reported_seq":330,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-20T03:36:24.267478+0000","last_change":"2023-10-20T03:36:22.234094+0000","last_active":"2023-10-20T03:36:24.267478+0000","last_peered":"2023-10-20T03:36:24.267478+0000","last_clean":"2023-10-20T03:36:24.267478+0000","last_became_active":"2023-10-20T03:36:15.158182+0000","last_became_peered":"2023-10-20T03:36:15.158182+0000","last_unstale":"2023-10-20T03:36:24.267478+0000","last_undegraded":"2023-10-20T03:36:24.267478+0000","last_fullsized":"2023-10-20T03:36:24.267478+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'23","last_scrub_stamp":"2023-10-20T03:36:18.357834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:36:18.357834+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:37:32.361242+0000","scrub_duration":0.010344032,"objects_trimmed":2,"snaptrim_duration":0.011728872,"stat_sum":{"num_bytes":786432,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12103,"num_write":21,"num_write_kb":6192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11,3,5,0,7],"acting":[6,1,11,3,5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"4"},{"start":"3b","length":"1"},{"start":"3f","length":"1"}]},{"pgid":"1.0","version":"26'157","reported_seq":239,"reported_epoch":101,"state":"active+clean","last_fresh":"2023-10-20T03:36:23.256831+0000","last_change":"2023-10-20T03:36:19.325301+0000","last_active":"2023-10-20T03:36:23.256831+0000","last_peered":"2023-10-20T03:36:23.256831+0000","last_clean":"2023-10-20T03:36:23.256831+0000","last_became_active":"2023-10-20T03:36:12.126368+0000","last_became_peered":"2023-10-20T03:36:12.126368+0000","last_unstale":"2023-10-20T03:36:23.256831+0000","last_undegraded":"2023-10-20T03:36:23.256831+0000","last_fullsized":"2023-10-20T03:36:23.256831+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'157","last_scrub_stamp":"2023-10-20T03:36:19.324962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:01.005741+0000","last_clean_scrub_stamp":"2023-10-20T03:36:19.324962+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:37:23.639536+0000","scrub_duration":0.0048410670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":86,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-20T03:36:24.262678+0000","last_change":"2023-10-20T03:35:05.117915+0000","last_active":"2023-10-20T03:36:24.262678+0000","last_peered":"2023-10-20T03:36:24.262678+0000","last_clean":"2023-10-20T03:36:24.262678+0000","last_became_active":"2023-10-20T03:35:05.117258+0000","last_became_peered":"2023-10-20T03:35:05.117258+0000","last_unstale":"2023-10-20T03:36:24.262678+0000","last_undegraded":"2023-10-20T03:36:24.262678+0000","last_fullsized":"2023-10-20T03:36:24.262678+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:32.359335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"100'96","reported_seq":390,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-20T03:36:25.274666+0000","last_change":"2023-10-20T03:36:25.276088+0000","last_active":"2023-10-20T03:36:25.274666+0000","last_peered":"2023-10-20T03:36:25.274666+0000","last_clean":"2023-10-20T03:36:25.274666+0000","last_became_active":"2023-10-20T03:35:14.221482+0000","last_became_peered":"2023-10-20T03:35:14.221482+0000","last_unstale":"2023-10-20T03:36:25.274666+0000","last_undegraded":"2023-10-20T03:36:25.274666+0000","last_fullsized":"2023-10-20T03:36:25.274666+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001895849,"stat_sum":{"num_bytes":7454720,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":17613,"num_write":122,"num_write_kb":9104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,0,4,8,10,7],"acting":[14,3,0,4,8,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"4"},{"start":"3b","length":"1"},{"start":"3f","length":"1"},{"start":"42","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":85,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-20T03:36:24.284333+0000","last_change":"2023-10-20T03:35:05.201208+0000","last_active":"2023-10-20T03:36:24.284333+0000","last_peered":"2023-10-20T03:36:24.284333+0000","last_clean":"2023-10-20T03:36:24.284333+0000","last_became_active":"2023-10-20T03:35:05.200512+0000","last_became_peered":"2023-10-20T03:35:05.200512+0000","last_unstale":"2023-10-20T03:36:24.284333+0000","last_undegraded":"2023-10-20T03:36:24.284333+0000","last_fullsized":"2023-10-20T03:36:24.284333+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:30.668122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"103'95","reported_seq":385,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-20T03:36:25.294656+0000","last_change":"2023-10-20T03:36:25.297238+0000","last_active":"2023-10-20T03:36:25.294656+0000","last_peered":"2023-10-20T03:36:25.294656+0000","last_clean":"2023-10-20T03:36:25.294656+0000","last_became_active":"2023-10-20T03:35:14.311165+0000","last_became_peered":"2023-10-20T03:35:14.311165+0000","last_unstale":"2023-10-20T03:36:25.294656+0000","last_undegraded":"2023-10-20T03:36:25.294656+0000","last_fullsized":"2023-10-20T03:36:25.294656+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:42.671288+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.023109547000000001,"stat_sum":{"num_bytes":5210112,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":18731,"num_write":31,"num_write_kb":7328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,14,10,7,3,2],"acting":[13,4,14,10,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"4"},{"start":"3b","length":"1"},{"start":"3f","length":"1"},{"start":"42","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":86,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-20T03:36:24.261884+0000","last_change":"2023-10-20T03:35:05.113211+0000","last_active":"2023-10-20T03:36:24.261884+0000","last_peered":"2023-10-20T03:36:24.261884+0000","last_clean":"2023-10-20T03:36:24.261884+0000","last_became_active":"2023-10-20T03:35:05.112592+0000","last_became_peered":"2023-10-20T03:35:05.112592+0000","last_unstale":"2023-10-20T03:36:24.261884+0000","last_undegraded":"2023-10-20T03:36:24.261884+0000","last_fullsized":"2023-10-20T03:36:24.261884+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:27.383114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1],"acting":[14,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"103'63","reported_seq":388,"reported_epoch":103,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:36:25.288409+0000","last_change":"2023-10-20T03:36:17.189168+0000","last_active":"2023-10-20T03:36:25.288409+0000","last_peered":"2023-10-20T03:36:25.288409+0000","last_clean":"2023-10-20T03:36:15.290892+0000","last_became_active":"2023-10-20T03:36:17.189168+0000","last_became_peered":"2023-10-20T03:36:17.189168+0000","last_unstale":"2023-10-20T03:36:25.288409+0000","last_undegraded":"2023-10-20T03:36:17.176423+0000","last_fullsized":"2023-10-20T03:36:17.172773+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.13851994500000001,"stat_sum":{"num_bytes":5931008,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":21937,"num_write":53,"num_write_kb":11536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":17907712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4,2,6,15,2147483647],"acting":[3,7,4,2,6,15,2147483647],"avail_no_missing":["3(0)","2(3)","4(2)","6(4)","7(1)","15(5)"],"object_location_counts":[{"shards":"2(3),3(0),4(2),6(4),7(1),15(5)","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":89,"reported_epoch":102,"state":"active+undersized","last_fresh":"2023-10-20T03:36:24.284541+0000","last_change":"2023-10-20T03:36:17.172853+0000","last_active":"2023-10-20T03:36:24.284541+0000","last_peered":"2023-10-20T03:36:24.284541+0000","last_clean":"2023-10-20T03:36:15.147401+0000","last_became_active":"2023-10-20T03:36:17.172853+0000","last_became_peered":"2023-10-20T03:36:17.172853+0000","last_unstale":"2023-10-20T03:36:24.284541+0000","last_undegraded":"2023-10-20T03:36:24.284541+0000","last_fullsized":"2023-10-20T03:36:17.171442+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8],"acting":[8],"avail_no_missing":["8"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"103'82","reported_seq":420,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-20T03:36:25.298528+0000","last_change":"2023-10-20T03:36:25.302421+0000","last_active":"2023-10-20T03:36:25.298528+0000","last_peered":"2023-10-20T03:36:25.298528+0000","last_clean":"2023-10-20T03:36:25.298528+0000","last_became_active":"2023-10-20T03:35:38.811204+0000","last_became_peered":"2023-10-20T03:35:38.811204+0000","last_unstale":"2023-10-20T03:36:25.298528+0000","last_undegraded":"2023-10-20T03:36:25.298528+0000","last_fullsized":"2023-10-20T03:36:25.298528+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'80","last_scrub_stamp":"2023-10-20T03:36:24.240577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:36:24.240577+0000","objects_scrubbed":4,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:37:30.222724+0000","scrub_duration":0.0094106320000000004,"objects_trimmed":1,"snaptrim_duration":0.014536751000000001,"stat_sum":{"num_bytes":6717440,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":37743,"num_write":43,"num_write_kb":13408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4866048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,1,0,11,2,14],"acting":[10,8,1,0,11,2,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"4"},{"start":"3b","length":"1"},{"start":"3f","length":"1"},{"start":"42","length":"1"}]},{"pgid":"3.5","version":"97'32","reported_seq":307,"reported_epoch":103,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:36:25.297970+0000","last_change":"2023-10-20T03:36:17.185985+0000","last_active":"2023-10-20T03:36:25.297970+0000","last_peered":"2023-10-20T03:36:25.297970+0000","last_clean":"2023-10-20T03:36:15.147049+0000","last_became_active":"2023-10-20T03:36:17.185985+0000","last_became_peered":"2023-10-20T03:36:17.185985+0000","last_unstale":"2023-10-20T03:36:25.297970+0000","last_undegraded":"2023-10-20T03:36:17.171750+0000","last_fullsized":"2023-10-20T03:36:17.171338+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:28.376769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085077720000000003,"stat_sum":{"num_bytes":802816,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":16,"num_read_kb":3604,"num_write":30,"num_write_kb":1664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,2,2147483647,1,11],"acting":[8,13,5,2,2147483647,1,11],"avail_no_missing":["8(0)","1(5)","2(3)","5(2)","11(6)","13(1)"],"object_location_counts":[{"shards":"1(5),2(3),5(2),8(0),11(6),13(1)","objects":2}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":86,"reported_epoch":102,"state":"active+undersized","last_fresh":"2023-10-20T03:36:24.262844+0000","last_change":"2023-10-20T03:36:17.187799+0000","last_active":"2023-10-20T03:36:24.262844+0000","last_peered":"2023-10-20T03:36:24.262844+0000","last_clean":"2023-10-20T03:36:15.148140+0000","last_became_active":"2023-10-20T03:36:17.187799+0000","last_became_peered":"2023-10-20T03:36:17.187799+0000","last_unstale":"2023-10-20T03:36:24.262844+0000","last_undegraded":"2023-10-20T03:36:24.262844+0000","last_fullsized":"2023-10-20T03:36:17.182506+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"102'8","reported_seq":294,"reported_epoch":103,"state":"active+undersized+degraded","last_fresh":"2023-10-20T03:36:25.277796+0000","last_change":"2023-10-20T03:36:24.285689+0000","last_active":"2023-10-20T03:36:25.277796+0000","last_peered":"2023-10-20T03:36:25.277796+0000","last_clean":"2023-10-20T03:36:15.253957+0000","last_became_active":"2023-10-20T03:36:17.192050+0000","last_became_peered":"2023-10-20T03:36:17.192050+0000","last_unstale":"2023-10-20T03:36:25.277796+0000","last_undegraded":"2023-10-20T03:36:24.281966+0000","last_fullsized":"2023-10-20T03:36:17.181362+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:35.807147+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.091105647999999997,"stat_sum":{"num_bytes":212992,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":291,"num_write":5,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":98304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,15,2147483647,2,6,10,13],"acting":[1,15,2147483647,2,6,10,13],"avail_no_missing":["1(0)","2(3)","6(4)","10(5)","13(6)","15(1)"],"object_location_counts":[{"shards":"1(0),2(3),6(4),10(5),13(6),15(1)","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-20T03:36:24.269819+0000","last_change":"2023-10-20T03:36:22.888238+0000","last_active":"2023-10-20T03:36:24.269819+0000","last_peered":"2023-10-20T03:36:24.269819+0000","last_clean":"2023-10-20T03:36:24.269819+0000","last_became_active":"2023-10-20T03:35:05.122056+0000","last_became_peered":"2023-10-20T03:35:05.122056+0000","last_unstale":"2023-10-20T03:36:24.269819+0000","last_undegraded":"2023-10-20T03:36:24.269819+0000","last_fullsized":"2023-10-20T03:36:24.269819+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:36:22.888060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:36:22.888060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:37:36.566098+0000","scrub_duration":0.0030881860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"102'15","reported_seq":337,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-20T03:36:25.296916+0000","last_change":"2023-10-20T03:36:25.298379+0000","last_active":"2023-10-20T03:36:25.296916+0000","last_peered":"2023-10-20T03:36:25.296916+0000","last_clean":"2023-10-20T03:36:25.296916+0000","last_became_active":"2023-10-20T03:36:15.163308+0000","last_became_peered":"2023-10-20T03:36:15.163308+0000","last_unstale":"2023-10-20T03:36:25.296916+0000","last_undegraded":"2023-10-20T03:36:25.296916+0000","last_fullsized":"2023-10-20T03:36:25.296916+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2023-10-20T03:36:22.914855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:36:22.914855+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:37:38.417746+0000","scrub_duration":0.0061573349999999999,"objects_trimmed":0,"snaptrim_duration":0.024037408999999999,"stat_sum":{"num_bytes":1785856,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":4757,"num_write":12,"num_write_kb":2544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1818624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,11,6,1,14,10,0],"acting":[13,11,6,1,14,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"4"},{"start":"3b","length":"1"},{"start":"3f","length":"1"},{"start":"42","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":102,"state":"active+undersized","last_fresh":"2023-10-20T03:36:24.250602+0000","last_change":"2023-10-20T03:36:17.183542+0000","last_active":"2023-10-20T03:36:24.250602+0000","last_peered":"2023-10-20T03:36:24.250602+0000","last_clean":"2023-10-20T03:35:04.079338+0000","last_became_active":"2023-10-20T03:36:17.183542+0000","last_became_peered":"2023-10-20T03:36:17.183542+0000","last_unstale":"2023-10-20T03:36:24.250602+0000","last_undegraded":"2023-10-20T03:36:24.250602+0000","last_fullsized":"2023-10-20T03:36:17.180839+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:35:04.079338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:33.501938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11],"acting":[11],"avail_no_missing":["11"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"100'90","reported_seq":528,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-10-20T03:36:25.297480+0000","last_change":"2023-10-20T03:36:25.298765+0000","last_active":"2023-10-20T03:36:25.297480+0000","last_peered":"2023-10-20T03:36:25.297480+0000","last_clean":"2023-10-20T03:36:25.297480+0000","last_became_active":"2023-10-20T03:36:15.165044+0000","last_became_peered":"2023-10-20T03:36:15.165044+0000","last_unstale":"2023-10-20T03:36:25.297480+0000","last_undegraded":"2023-10-20T03:36:25.297480+0000","last_fullsized":"2023-10-20T03:36:25.297480+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:13.171293+0000","last_clean_scrub_stamp":"2023-10-20T03:35:13.171293+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:36:41.695149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024118522999999999,"stat_sum":{"num_bytes":6946816,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":37331,"num_write":135,"num_write_kb":10320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":38027264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14,6,2,4,0,15],"acting":[13,14,6,2,4,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"3"},{"start":"6","length":"2"},{"start":"9","length":"5"},{"start":"f","length":"1"},{"start":"16","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"3"},{"start":"21","length":"1"},{"start":"23","length":"1"},{"start":"26","length":"3"},{"start":"2b","length":"1"},{"start":"2e","length":"1"},{"start":"30","length":"2"},{"start":"33","length":"1"},{"start":"36","length":"4"},{"start":"3b","length":"1"},{"start":"3f","length":"1"},{"start":"42","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":93,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-10-20T03:36:24.284708+0000","last_change":"2023-10-20T03:36:24.084856+0000","last_active":"2023-10-20T03:36:24.284708+0000","last_peered":"2023-10-20T03:36:24.284708+0000","last_clean":"2023-10-20T03:36:24.284708+0000","last_became_active":"2023-10-20T03:35:05.198615+0000","last_became_peered":"2023-10-20T03:35:05.198615+0000","last_unstale":"2023-10-20T03:36:24.284708+0000","last_undegraded":"2023-10-20T03:36:24.284708+0000","last_fullsized":"2023-10-20T03:36:24.284708+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-20T03:36:24.084419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-20T03:35:04.079338+0000","last_clean_scrub_stamp":"2023-10-20T03:36:24.084419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-20T03:37:35.600948+0000","scrub_duration":0.0041413509999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":87080960,"num_objects":55,"num_object_clones":17,"num_object_copies":385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":2,"num_read":420,"num_read_kb":258144,"num_write":944,"num_write_kb":137984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":91504640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":152768512,"data_stored":188514304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1038,"ondisk_log_size":1038,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":13,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":1119296,"data_compressed":16060,"data_compressed_allocated":69632,"data_compressed_original":1101824,"omap_allocated":2350,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280532,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42776,"kb_used_data":11596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329064,"statfs":{"total":96636764160,"available":96592961536,"internally_reserved":0,"allocated":11874304,"data_stored":12120701,"data_compressed":161749,"data_compressed_allocated":331776,"data_compressed_original":663552,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":11.994,"5min":11.994,"15min":11.994},"last":0.93000000000000005}]},{"osd":1,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":1.5580000000000001}]},{"osd":2,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":2.0179999999999998,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.88300000000000001}]},{"osd":3,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":2.403,"5min":2.403,"15min":2.403},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.825,"5min":1.825,"15min":1.825},"last":1.0900000000000001}]},{"osd":4,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.748,"5min":1.748,"15min":1.748},"last":1.151},{"interface":"front","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.393}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5069999999999999}]},{"osd":7,"last update":"Fri Oct 20 03:36:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":2.3719999999999999,"5min":2.3719999999999999,"15min":2.3719999999999999},"last":1.615},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.536}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8600000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7430000000000001}]},{"osd":10,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"max":{"1min":2.4510000000000001,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":1.6579999999999999},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.6599999999999999},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.7330000000000001,"5min":2.7330000000000001,"15min":2.7330000000000001},"last":1.6739999999999999}]},{"osd":11,"last update":"Fri Oct 20 03:36:17 2023","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"max":{"1min":2.4940000000000002,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.802},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":2.6869999999999998,"5min":2.6869999999999998,"15min":2.6869999999999998},"last":1.7310000000000001}]},{"osd":12,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.476},{"interface":"front","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":1.7509999999999999}]},{"osd":14,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":2.5710000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.823,"5min":2.823,"15min":2.823},"last":1.5880000000000001}]},{"osd":15,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":1.679,"15min":1.679},"min":{"1min":1.038,"5min":1.038,"15min":1.038},"max":{"1min":2.403,"5min":2.403,"15min":2.403},"last":1.847},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.7759999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.7729999999999999}]}]},{"osd":15,"up_from":22,"seq":94489280533,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35820,"kb_used_data":8792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336020,"statfs":{"total":96636764160,"available":96600084480,"internally_reserved":0,"allocated":9003008,"data_stored":14236429,"data_compressed":167257,"data_compressed_allocated":360448,"data_compressed_original":1167360,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.21},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.913,"5min":1.913,"15min":1.913},"last":1.111}]},{"osd":1,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.79400000000000004}]},{"osd":2,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.996,"5min":0.996,"15min":0.996},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":0.874}]},{"osd":3,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.70599999999999996}]},{"osd":4,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":1.177},{"interface":"front","average":{"1min":1.359,"5min":1.359,"15min":1.359},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.0630000000000002},"last":1.3009999999999999}]},{"osd":5,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.825,"5min":1.825,"15min":1.825},"last":1.2470000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]},{"osd":7,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.956,"5min":1.956,"15min":1.956},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.3700000000000001}]},{"osd":8,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.3399999999999999,"5min":2.3399999999999999,"15min":2.3399999999999999},"last":1.337},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"min":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"max":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"last":1.401}]},{"osd":10,"last update":"Fri Oct 20 03:36:13 2023","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.427,"15min":1.427},"min":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"min":{"1min":1.137,"5min":1.137,"15min":1.137},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.4319999999999999}]},{"osd":14,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":2.8849999999999998,"5min":2.8849999999999998,"15min":2.8849999999999998},"last":1.147},{"interface":"front","average":{"1min":1.595,"5min":1.595,"15min":1.595},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":3.004,"5min":3.004,"15min":3.004},"last":1.4630000000000001}]}]},{"osd":14,"up_from":22,"seq":94489280533,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44328,"kb_used_data":13148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327512,"statfs":{"total":96636764160,"available":96591372288,"internally_reserved":0,"allocated":13463552,"data_stored":13886077,"data_compressed":161749,"data_compressed_allocated":331776,"data_compressed_original":663552,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,7,8,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.034,"5min":1.034,"15min":1.034},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.60099999999999998}]},{"osd":1,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.04,"15min":1.04},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":1.4990000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":3,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.82899999999999996}]},{"osd":4,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.415}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4470000000000001}]},{"osd":7,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.304,"5min":1.304,"15min":1.304},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":1.1659999999999999}]},{"osd":8,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.4199999999999999,"5min":2.4199999999999999,"15min":2.4199999999999999},"last":1.6180000000000001}]},{"osd":10,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.526,"5min":1.526,"15min":1.526},"last":0.996}]},{"osd":11,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.2999999999999998,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":2.6589999999999998,"5min":2.6589999999999998,"15min":2.6589999999999998},"last":0.90700000000000003}]},{"osd":13,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":2.4900000000000002,"5min":2.4900000000000002,"15min":2.4900000000000002},"last":1.095},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"max":{"1min":3.266,"5min":3.266,"15min":3.266},"last":1.3700000000000001}]},{"osd":15,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.5409999999999999,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":3.2090000000000001,"5min":3.2090000000000001,"15min":3.2090000000000001},"last":1.8129999999999999}]}]},{"osd":5,"up_from":21,"seq":90194313237,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32988,"kb_used_data":5968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338852,"statfs":{"total":96636764160,"available":96602984448,"internally_reserved":0,"allocated":6111232,"data_stored":5850787,"data_compressed":159718,"data_compressed_allocated":327680,"data_compressed_original":655360,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,6,7,8,10,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.878,"15min":0.878},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.91500000000000004}]},{"osd":3,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.798,"5min":1.798,"15min":1.798},"last":1.3959999999999999}]},{"osd":4,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.528,"5min":1.528,"15min":1.528},"last":1.151},{"interface":"front","average":{"1min":1.052,"5min":1.052,"15min":1.052},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":1.2110000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]},{"osd":7,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.2759999999999998,"5min":2.2759999999999998,"15min":2.2759999999999998},"last":0.64800000000000002}]},{"osd":8,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.036},{"interface":"front","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":1.6459999999999999}]},{"osd":9,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.351,"15min":1.351},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.766,"5min":2.766,"15min":2.766},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":2.484,"5min":2.484,"15min":2.484},"last":1.351}]},{"osd":10,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.7000000000000002,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":1.9370000000000001,"5min":1.9370000000000001,"15min":1.9370000000000001},"last":1.2589999999999999}]},{"osd":11,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"last":1.6919999999999999}]},{"osd":12,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"max":{"1min":2.6120000000000001,"5min":2.6120000000000001,"15min":2.6120000000000001},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"min":{"1min":1,"5min":1,"15min":1},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.4890000000000001}]},{"osd":15,"last update":"Fri Oct 20 03:36:21 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":1.026,"5min":1.026,"15min":1.026},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.538,"15min":1.538},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.841}]}]},{"osd":0,"up_from":22,"seq":94489280532,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44472,"kb_used_data":13292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327368,"statfs":{"total":96636764160,"available":96591224832,"internally_reserved":0,"allocated":13611008,"data_stored":13546109,"data_compressed":161749,"data_compressed_allocated":331776,"data_compressed_original":663552,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,3,5,7,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":0.68400000000000005}]},{"osd":2,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.091},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":0.70499999999999996}]},{"osd":3,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.95999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":5,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.749}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":7,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.0110000000000001,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":0.86499999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.681}]},{"osd":9,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":0.995},{"interface":"front","average":{"1min":1.351,"5min":1.351,"15min":1.351},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":2.794,"5min":2.794,"15min":2.794},"last":1.2030000000000001}]},{"osd":11,"last update":"Fri Oct 20 03:36:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.879,"5min":1.879,"15min":1.879},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.2130000000000001}]},{"osd":12,"last update":"Fri Oct 20 03:36:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":1.387},{"interface":"front","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":1.486}]},{"osd":13,"last update":"Fri Oct 20 03:36:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":1.306},{"interface":"front","average":{"1min":1.327,"5min":1.327,"15min":1.327},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":0.88900000000000001}]},{"osd":14,"last update":"Fri Oct 20 03:36:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1.018},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.153}]},{"osd":15,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":2.2789999999999999,"5min":2.2789999999999999,"15min":2.2789999999999999},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":1.0640000000000001}]}]},{"osd":1,"up_from":22,"seq":94489280533,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":10024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":10264576,"data_stored":10199677,"data_compressed":161749,"data_compressed_allocated":331776,"data_compressed_original":663552,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":5.6740000000000004,"5min":5.6740000000000004,"15min":5.6740000000000004},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":0.64500000000000002}]},{"osd":2,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.999,"15min":0.999},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.80000000000000004}]},{"osd":3,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":0.59799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":5,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.6440000000000001,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":0.50800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":7,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.351,"5min":1.351,"15min":1.351},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":1.01}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":9,"last update":"Fri Oct 20 03:36:06 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":2.085,"5min":2.085,"15min":2.085},"last":4.5659999999999998},{"interface":"front","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":6.8940000000000001}]},{"osd":10,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.665,"5min":1.665,"15min":1.665},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":2.7000000000000002,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":3.004,"5min":3.004,"15min":3.004},"last":1.206}]},{"osd":11,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":2.0030000000000001,"5min":2.0030000000000001,"15min":2.0030000000000001},"min":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"max":{"1min":6.367,"5min":6.367,"15min":6.367},"last":1.161},{"interface":"front","average":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.0329999999999999},"min":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":6.1109999999999998,"5min":6.1109999999999998,"15min":6.1109999999999998},"last":0.996}]},{"osd":12,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.603,"15min":1.603},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":2.6760000000000002,"5min":2.6760000000000002,"15min":2.6760000000000002},"last":1.0740000000000001}]},{"osd":13,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.825,"15min":1.825},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":5.8799999999999999,"5min":5.8799999999999999,"15min":5.8799999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":2.0430000000000001,"5min":2.0430000000000001,"15min":2.0430000000000001},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":6.4619999999999997,"5min":6.4619999999999997,"15min":6.4619999999999997},"last":1.0600000000000001}]},{"osd":15,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":3.056,"5min":3.056,"15min":3.056},"last":1.232},{"interface":"front","average":{"1min":1.99,"5min":1.99,"15min":1.99},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":6.7990000000000004,"5min":6.7990000000000004,"15min":6.7990000000000004},"last":1.0449999999999999}]}]},{"osd":2,"up_from":22,"seq":94489280532,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41140,"kb_used_data":9960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330700,"statfs":{"total":96636764160,"available":96594636800,"internally_reserved":0,"allocated":10199040,"data_stored":11207293,"data_compressed":161749,"data_compressed_allocated":331776,"data_compressed_original":663552,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,10,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":1.163},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.79100000000000004}]},{"osd":1,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":0.997}]},{"osd":3,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.127}]},{"osd":4,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":1.27},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":1.3100000000000001}]},{"osd":5,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.22}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":7,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.613}]},{"osd":8,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.333,"15min":1.333},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.3679999999999999,"5min":2.3679999999999999,"15min":2.3679999999999999},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":3.1040000000000001,"5min":3.1040000000000001,"15min":3.1040000000000001},"last":1.661}]},{"osd":9,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.573,"5min":2.573,"15min":2.573},"last":1.288},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.393}]},{"osd":10,"last update":"Fri Oct 20 03:36:24 2023","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":1.577},{"interface":"front","average":{"1min":1.351,"5min":1.351,"15min":1.351},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.7}]},{"osd":15,"last update":"Fri Oct 20 03:36:04 2023","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":1.7110000000000001,"15min":1.7110000000000001},"min":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"max":{"1min":2.4129999999999998,"5min":2.4129999999999998,"15min":2.4129999999999998},"last":1.9339999999999999},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.7849999999999999}]}]},{"osd":3,"up_from":22,"seq":94489280532,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63580,"kb_used_data":14344,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94308260,"statfs":{"total":96636764160,"available":96571658240,"internally_reserved":0,"allocated":14688256,"data_stored":19982573,"data_compressed":159718,"data_compressed_allocated":327680,"data_compressed_original":655360,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7,8,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":1,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.669,"5min":2.669,"15min":2.669},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.72399999999999998}]},{"osd":2,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.747}]},{"osd":4,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.76400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":7,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.9239999999999999,"5min":2.9239999999999999,"15min":2.9239999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.556,"5min":1.556,"15min":1.556},"last":0.95099999999999996}]},{"osd":8,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.90900000000000003}]},{"osd":9,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.622},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.7450000000000001,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.63400000000000001}]},{"osd":10,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.343,"15min":1.343},"min":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"max":{"1min":2.1680000000000001,"5min":2.1680000000000001,"15min":2.1680000000000001},"last":1.238},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":1.756,"5min":1.756,"15min":1.756},"last":1.3120000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":14,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":2.5859999999999999,"5min":2.5859999999999999,"15min":2.5859999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.4510000000000001,"15min":1.4510000000000001},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.1619999999999999}]},{"osd":15,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.0430000000000001,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":1.0740000000000001}]}]},{"osd":4,"up_from":22,"seq":94489280533,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67296,"kb_used_data":18068,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94304544,"statfs":{"total":96636764160,"available":96567853056,"internally_reserved":0,"allocated":18501632,"data_stored":23569021,"data_compressed":161749,"data_compressed_allocated":331776,"data_compressed_original":663552,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5,6,8,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":3,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.236}]},{"osd":5,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.4930000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]},{"osd":8,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.347},{"interface":"front","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.3919999999999999,"5min":2.3919999999999999,"15min":2.3919999999999999},"last":1.8340000000000001}]},{"osd":9,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.54},{"interface":"front","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":3.2269999999999999,"5min":3.2269999999999999,"15min":3.2269999999999999},"last":1.4039999999999999}]},{"osd":10,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":1.6599999999999999,"15min":1.6599999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":6.7599999999999998,"5min":6.7599999999999998,"15min":6.7599999999999998},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.407,"5min":2.407,"15min":2.407},"last":1.5329999999999999}]},{"osd":11,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":3.3479999999999999,"5min":3.3479999999999999,"15min":3.3479999999999999},"last":1.9039999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.4529999999999998,"5min":2.4529999999999998,"15min":2.4529999999999998},"last":1.5800000000000001}]},{"osd":12,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":2.589,"5min":2.589,"15min":2.589},"last":1.95},{"interface":"front","average":{"1min":1.405,"5min":1.405,"15min":1.405},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":3.3090000000000002,"5min":3.3090000000000002,"15min":3.3090000000000002},"last":1.712}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.863}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.873}]}]},{"osd":6,"up_from":89,"seq":382252089348,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38684,"kb_used_data":10316,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94333156,"statfs":{"total":96636764160,"available":96597151744,"internally_reserved":0,"allocated":10563584,"data_stored":14954270,"data_compressed":166205,"data_compressed_allocated":360448,"data_compressed_original":1236992,"omap_allocated":6749,"internal_metadata":29025699},"hb_peers":[0,1,3,5,7,8,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6850000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4350000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.643}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.718}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.788}]}]},{"osd":7,"up_from":22,"seq":94489280531,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34912,"kb_used_data":7892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336928,"statfs":{"total":96636764160,"available":96601014272,"internally_reserved":0,"allocated":8081408,"data_stored":8913533,"data_compressed":161749,"data_compressed_allocated":331776,"data_compressed_original":663552,"omap_allocated":1588,"internal_metadata":27654604},"hb_peers":[0,2,3,4,5,6,8,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":2,"last update":"Fri Oct 20 03:36:14 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":1.2170000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.94599999999999995}]},{"osd":3,"last update":"Fri Oct 20 03:36:14 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":1.107}]},{"osd":4,"last update":"Fri Oct 20 03:36:14 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":1.0309999999999999}]},{"osd":5,"last update":"Fri Oct 20 03:36:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.623},{"interface":"front","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.76600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]},{"osd":8,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.3100000000000001}]},{"osd":9,"last update":"Fri Oct 20 03:36:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":1.171}]},{"osd":10,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":1.25},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":1.4259999999999999}]},{"osd":11,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.401},{"interface":"front","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":1.4550000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.377}]},{"osd":14,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.359,"15min":1.359},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.677,"5min":1.677,"15min":1.677},"last":1.516},{"interface":"front","average":{"1min":1.496,"5min":1.496,"15min":1.496},"min":{"1min":1.022,"5min":1.022,"15min":1.022},"max":{"1min":1.9670000000000001,"5min":1.9670000000000001,"15min":1.9670000000000001},"last":1.3500000000000001}]}]},{"osd":8,"up_from":22,"seq":94489280532,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37216,"kb_used_data":10200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334624,"statfs":{"total":96636764160,"available":96598654976,"internally_reserved":0,"allocated":10444800,"data_stored":15509741,"data_compressed":159718,"data_compressed_allocated":327680,"data_compressed_original":655360,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,5,6,7,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.65900000000000003}]},{"osd":2,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":3.0920000000000001,"5min":3.0920000000000001,"15min":3.0920000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.55500000000000005}]},{"osd":3,"last update":"Fri Oct 20 03:36:05 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":3.2189999999999999,"5min":3.2189999999999999,"15min":3.2189999999999999},"last":0.94599999999999995}]},{"osd":5,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.196,"15min":1.196},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":2.8450000000000002,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.78300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":7,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":4.298,"5min":4.298,"15min":4.298},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.9399999999999999,"5min":2.9399999999999999,"15min":2.9399999999999999},"last":0.995}]},{"osd":9,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.149,"15min":1.149},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.2669999999999999}]},{"osd":10,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":3.02,"5min":3.02,"15min":3.02},"last":0.82099999999999995}]},{"osd":11,"last update":"Fri Oct 20 03:36:15 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.476,"15min":1.476},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.218,"5min":2.218,"15min":2.218},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.726,"5min":1.726,"15min":1.726},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":5.141,"5min":5.141,"15min":5.141},"last":0.76600000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":13,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.25,"5min":2.25,"15min":2.25},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.7429999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":0.97199999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]},{"osd":15,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":3.2669999999999999,"5min":3.2669999999999999,"15min":3.2669999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":4.569,"5min":4.569,"15min":4.569},"last":0.88200000000000001}]}]},{"osd":9,"up_from":22,"seq":94489280529,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34732,"kb_used_data":7712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337108,"statfs":{"total":96636764160,"available":96601198592,"internally_reserved":0,"allocated":7897088,"data_stored":9760977,"data_compressed":131269,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1598,"internal_metadata":27654594},"hb_peers":[2,3,4,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":2,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":4.2530000000000001,"5min":4.2530000000000001,"15min":4.2530000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":2.6299999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":0.499}]},{"osd":3,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.9790000000000001,"5min":3.9790000000000001,"15min":3.9790000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":5.0789999999999997,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":0.51700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":8,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.7429999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":3.6659999999999999,"5min":3.6659999999999999,"15min":3.6659999999999999},"last":1.196}]},{"osd":10,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":5.7859999999999996,"5min":5.7859999999999996,"15min":5.7859999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.679,"5min":1.679,"15min":1.679},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":6.9870000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":1.252}]},{"osd":11,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.476,"15min":1.476},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":5.1559999999999997,"5min":5.1559999999999997,"15min":5.1559999999999997},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":5.2999999999999998,"5min":5.2999999999999998,"15min":5.2999999999999998},"last":0.999}]},{"osd":12,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":4.9820000000000002,"5min":4.9820000000000002,"15min":4.9820000000000002},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.327,"5min":1.327,"15min":1.327},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.8199999999999998,"5min":2.8199999999999998,"15min":2.8199999999999998},"last":0.63100000000000001}]},{"osd":13,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":8.6479999999999997,"5min":8.6479999999999997,"15min":8.6479999999999997},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":1.8999999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":8.1170000000000009,"5min":8.1170000000000009,"15min":8.1170000000000009},"last":0.57099999999999995}]},{"osd":14,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":5.5199999999999996,"5min":5.5199999999999996,"15min":5.5199999999999996},"last":0.879},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":3.1320000000000001,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":0.72299999999999998}]},{"osd":15,"last update":"Fri Oct 20 03:36:03 2023","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.798,"15min":1.798},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":4.7919999999999998,"5min":4.7919999999999998,"15min":4.7919999999999998},"last":1.409},{"interface":"front","average":{"1min":1.738,"5min":1.738,"15min":1.738},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":5.6989999999999998,"5min":5.6989999999999998,"15min":5.6989999999999998},"last":1.0409999999999999}]}]},{"osd":10,"up_from":22,"seq":94489280531,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44936,"kb_used_data":13748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326904,"statfs":{"total":96636764160,"available":96590749696,"internally_reserved":0,"allocated":14077952,"data_stored":20277485,"data_compressed":159718,"data_compressed_allocated":327680,"data_compressed_original":655360,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,2,3,8,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"last":1.081},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.57799999999999996}]},{"osd":1,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.528,"5min":2.528,"15min":2.528},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":4.7679999999999998,"5min":4.7679999999999998,"15min":4.7679999999999998},"last":0.64900000000000002}]},{"osd":2,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.343},{"interface":"front","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.3049999999999999}]},{"osd":3,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.9019999999999999,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":0.96699999999999997}]},{"osd":8,"last update":"Fri Oct 20 03:36:13 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":1.212},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"last":0.86299999999999999}]},{"osd":9,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.464,"15min":1.464},"min":{"1min":1.071,"5min":1.071,"15min":1.071},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.78,"5min":1.78,"15min":1.78},"last":1.641}]},{"osd":11,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.171},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"max":{"1min":2.367,"5min":2.367,"15min":2.367},"last":0.91500000000000004}]},{"osd":12,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"min":{"1min":1.115,"5min":1.115,"15min":1.115},"max":{"1min":3.8559999999999999,"5min":3.8559999999999999,"15min":3.8559999999999999},"last":1.4650000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":1.391}]},{"osd":13,"last update":"Fri Oct 20 03:36:01 2023","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":2.8260000000000001,"5min":2.8260000000000001,"15min":2.8260000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"min":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"max":{"1min":3.7069999999999999,"5min":3.7069999999999999,"15min":3.7069999999999999},"last":1.26}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.425}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5649999999999999}]}]},{"osd":11,"up_from":22,"seq":94489280533,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43400,"kb_used_data":12212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328440,"statfs":{"total":96636764160,"available":96592322560,"internally_reserved":0,"allocated":12505088,"data_stored":14136974,"data_compressed":159718,"data_compressed_allocated":327680,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":4.1379999999999999,"5min":4.1379999999999999,"15min":4.1379999999999999},"last":0.91200000000000003}]},{"osd":2,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":4.2290000000000001,"5min":4.2290000000000001,"15min":4.2290000000000001},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Oct 20 03:36:13 2023","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":5.681,"5min":5.681,"15min":5.681},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.782,"5min":1.782,"15min":1.782},"last":0.72399999999999998}]},{"osd":4,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":5.8070000000000004,"5min":5.8070000000000004,"15min":5.8070000000000004},"last":1.234},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.111}]},{"osd":5,"last update":"Fri Oct 20 03:36:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":5.7190000000000003,"5min":5.7190000000000003,"15min":5.7190000000000003},"last":1.139}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":10,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":1.355},{"interface":"front","average":{"1min":1.631,"5min":1.631,"15min":1.631},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":7.2610000000000001,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":1.278}]},{"osd":12,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":9.1959999999999997,"5min":9.1959999999999997,"15min":9.1959999999999997},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":1.1659999999999999}]},{"osd":13,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":1.02,"5min":1.02,"15min":1.02},"max":{"1min":4.258,"5min":4.258,"15min":4.258},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.643,"5min":1.643,"15min":1.643},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":5.8360000000000003,"5min":5.8360000000000003,"15min":5.8360000000000003},"last":1.3839999999999999}]},{"osd":14,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.544,"15min":1.544},"min":{"1min":1.131,"5min":1.131,"15min":1.131},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.258},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":1.161,"5min":1.161,"15min":1.161},"max":{"1min":4.2830000000000004,"5min":4.2830000000000004,"15min":4.2830000000000004},"last":1.3280000000000001}]},{"osd":15,"last update":"Fri Oct 20 03:36:02 2023","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.7709999999999999,"15min":1.7709999999999999},"min":{"1min":1.179,"5min":1.179,"15min":1.179},"max":{"1min":5.7809999999999997,"5min":5.7809999999999997,"15min":5.7809999999999997},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":1.21}]}]},{"osd":12,"up_from":0,"seq":94489280531,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":8521,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":7539,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12419072,"data_stored":12644352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9072640,"data_stored":9297920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9007104,"data_stored":10305536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13041664,"data_stored":18624512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":17309696,"data_stored":22667264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4935680,"data_stored":4960256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9453568,"data_stored":13705216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6889472,"data_stored":8011776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":9265152,"data_stored":13524992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":6889472,"data_stored":9019392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":12419072,"data_stored":17190912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":11329536,"data_stored":11530240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":10682368,"data_stored":11218944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12271616,"data_stored":12984320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":7782400,"data_stored":12828672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-20T03:36:29.373 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2023-10-20T03:36:29.373 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2023-10-20T03:36:29.374 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:36:29.360+0000 7f5261bac640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:36:29.464 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:36:29.463+0000 7fd15ef99640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:36:29.524 INFO:teuthology.orchestra.run.smithi040.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-10-20T03:36:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-20T03:36:29.527 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-10-20T03:36:29.574 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:36:29.572+0000 7f106fd35640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:36:29.613 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-10-20T03:36:29.637 INFO:teuthology.orchestra.run.smithi040.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-10-20T03:36:29.640 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-20T03:36:29.641 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-10-20T03:36:29.665 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:36:29.663+0000 7f80e1211640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:36:29.757 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0: {} 2023-10-20T03:36:29.757 INFO:teuthology.orchestra.run.smithi033.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-10-20T03:36:29.765 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-10-20T03:36:29.776 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1: {} 2023-10-20T03:36:29.776 INFO:teuthology.orchestra.run.smithi033.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-10-20T03:36:29.795 INFO:teuthology.orchestra.run.smithi040.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-10-20T03:36:29.797 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2: {} 2023-10-20T03:36:29.797 INFO:teuthology.orchestra.run.smithi033.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-10-20T03:36:29.798 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-20T03:36:29.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-10-20T03:36:29.818 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3: {} 2023-10-20T03:36:29.819 INFO:teuthology.orchestra.run.smithi033.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-10-20T03:36:29.836 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4: {} 2023-10-20T03:36:29.837 INFO:teuthology.orchestra.run.smithi033.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-10-20T03:36:29.855 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5: {} 2023-10-20T03:36:29.856 INFO:teuthology.orchestra.run.smithi033.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-10-20T03:36:29.868 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:29.868+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:29.876 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6: {} 2023-10-20T03:36:29.876 INFO:teuthology.orchestra.run.smithi033.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-10-20T03:36:29.897 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7: {} 2023-10-20T03:36:29.897 INFO:teuthology.orchestra.run.smithi033.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-10-20T03:36:29.918 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8: {} 2023-10-20T03:36:29.919 INFO:teuthology.orchestra.run.smithi033.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-10-20T03:36:29.919 INFO:teuthology.orchestra.run.smithi033.stderr:Error ENXIO: problem getting command descriptions from osd.9 2023-10-20T03:36:29.919 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-10-20T03:36:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2023-10-20T03:36:29.920 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2023-10-20T03:36:29.920 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-10-20T03:36:29.933 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-10-20T03:36:29.941 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10: {} 2023-10-20T03:36:29.942 INFO:teuthology.orchestra.run.smithi033.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-10-20T03:36:29.964 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11: {} 2023-10-20T03:36:29.964 INFO:teuthology.orchestra.run.smithi033.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-10-20T03:36:29.969 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:36:29.967+0000 7f201c773640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:36:29.982 INFO:teuthology.orchestra.run.smithi033.stdout:osd.12: {} 2023-10-20T03:36:29.982 INFO:teuthology.orchestra.run.smithi033.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-10-20T03:36:29.999 INFO:teuthology.orchestra.run.smithi033.stderr:2023-10-20T03:36:29.996+0000 7fc29f7fe640 0 -- 172.21.15.33:0/4103818471 >> [v2:172.21.15.40:6812/3041424233,v1:172.21.15.40:6816/3041424233] conn(0x7fc2a021a7e0 msgr2=0x7fc2a021cbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-10-20T03:36:30.000 INFO:teuthology.orchestra.run.smithi033.stdout:osd.13: {} 2023-10-20T03:36:30.000 INFO:teuthology.orchestra.run.smithi033.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-10-20T03:36:30.022 INFO:teuthology.orchestra.run.smithi033.stdout:osd.14: {} 2023-10-20T03:36:30.022 INFO:teuthology.orchestra.run.smithi033.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-10-20T03:36:30.044 INFO:teuthology.orchestra.run.smithi033.stdout:osd.15: {} 2023-10-20T03:36:30.044 INFO:teuthology.orchestra.run.smithi033.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-10-20T03:36:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-10-20T03:36:30.049 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-10-20T03:36:30.051 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:30.051 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:30.066 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-10-20T03:36:30.069 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-10-20T03:36:30.171 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:36:30.164+0000 7f5419c85640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:36:30.189 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:30.190 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:30.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-10-20T03:36:30.272 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:36:30.268+0000 7fe4ce7d3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:36:30.312 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'unique_pool_0' removed 2023-10-20T03:36:30.312 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2023-10-20T03:36:30.339 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2023-10-20T03:36:30.348 INFO:tasks.thrashosds:joining thrashosds 2023-10-20T03:36:30.349 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:30.349 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:30.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-10-20T03:36:30.479 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:30.479 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:30.493 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-10-20T03:36:30.612 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:30.612 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:30.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-10-20T03:36:30.749 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:30.749 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:30.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-10-20T03:36:30.892 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:30.892 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:30.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-10-20T03:36:31.032 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:31.033 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:31.047 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-10-20T03:36:31.165 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:31.166 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:31.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-10-20T03:36:31.307 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:31.308 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:31.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-10-20T03:36:31.445 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:31.445 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:31.459 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-10-20T03:36:31.592 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:31.592 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:31.607 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-10-20T03:36:31.728 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:31.729 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:31.742 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-10-20T03:36:31.864 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:31.864 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:31.884 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-10-20T03:36:31.998 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:31.999 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:32.013 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-10-20T03:36:32.130 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:32.130 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:32.144 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-10-20T03:36:32.269 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:32.269 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:32.283 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-10-20T03:36:32.350 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-10-20T03:36:32.403 INFO:tasks.thrashosds.thrasher:inning osd 2023-10-20T03:36:32.404 INFO:tasks.thrashosds.thrasher:Adding osd 12 2023-10-20T03:36:32.404 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 12 2023-10-20T03:36:32.405 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:32.405 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:32.418 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-10-20T03:36:32.539 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:32.539 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:32.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-10-20T03:36:32.675 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:32.676 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:32.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-10-20T03:36:32.716 INFO:tasks.ceph.mon.a.smithi033.stderr:2023-10-20T03:36:32.712+0000 7f44d76b2640 -1 mon.a@0(leader).osd e107 definitely_dead 0 2023-10-20T03:36:32.811 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:32.812 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:32.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-10-20T03:36:32.947 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:32.947 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:32.961 DEBUG:teuthology.orchestra.run.smithi170:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-10-20T03:36:33.081 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:33.082 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:33.095 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-10-20T03:36:33.229 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:33.230 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:33.244 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-10-20T03:36:33.320 INFO:teuthology.orchestra.run.smithi033.stderr:noscrub is unset 2023-10-20T03:36:33.320 INFO:teuthology.orchestra.run.smithi033.stderr:marked in osd.12. 2023-10-20T03:36:33.345 INFO:tasks.thrashosds.thrasher:Added osd 12 2023-10-20T03:36:33.345 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2023-10-20T03:36:33.348 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-10-20T03:36:33.371 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:33.371 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:33.385 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-10-20T03:36:33.477 INFO:teuthology.orchestra.run.smithi033.stdout:mon.a: {} 2023-10-20T03:36:33.477 INFO:teuthology.orchestra.run.smithi033.stderr:mon.a: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2023-10-20T03:36:33.489 INFO:teuthology.orchestra.run.smithi033.stdout:mon.b: {} 2023-10-20T03:36:33.489 INFO:teuthology.orchestra.run.smithi033.stderr:mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2023-10-20T03:36:33.502 INFO:teuthology.orchestra.run.smithi033.stdout:mon.c: {} 2023-10-20T03:36:33.502 INFO:teuthology.orchestra.run.smithi033.stderr:mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2023-10-20T03:36:33.512 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2023-10-20T03:36:33.512 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-10-20T03:36:33.519 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-10-20T03:36:34.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~15s 2023-10-20T03:36:34.323 INFO:teuthology.orchestra.run.smithi033.stderr:nodeep-scrub is unset 2023-10-20T03:36:34.324 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.0 to 1 (10000) 2023-10-20T03:36:34.349 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2023-10-20T03:36:35.336 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.0 primary-affinity to 1 (8655362) 2023-10-20T03:36:35.363 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-10-20T03:36:36.350 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.4 to 1 (10000) 2023-10-20T03:36:36.378 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2023-10-20T03:36:37.362 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.4 primary-affinity to 1 (8655362) 2023-10-20T03:36:37.390 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2023-10-20T03:36:38.381 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.8 to 1 (10000) 2023-10-20T03:36:38.405 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2023-10-20T03:36:39.384 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.8 primary-affinity to 1 (8655362) 2023-10-20T03:36:39.412 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2023-10-20T03:36:40.397 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.12 to 1 (10000) 2023-10-20T03:36:40.425 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2023-10-20T03:36:41.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~22s 2023-10-20T03:36:41.418 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.12 primary-affinity to 1 (8655362) 2023-10-20T03:36:41.446 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-10-20T03:36:42.422 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.1 to 1 (10000) 2023-10-20T03:36:42.447 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2023-10-20T03:36:43.436 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.1 primary-affinity to 1 (8655362) 2023-10-20T03:36:43.463 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2023-10-20T03:36:44.448 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.5 to 1 (10000) 2023-10-20T03:36:44.476 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2023-10-20T03:36:45.459 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.5 primary-affinity to 1 (8655362) 2023-10-20T03:36:45.487 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2023-10-20T03:36:46.472 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.9 to 1 (10000) 2023-10-20T03:36:46.495 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2023-10-20T03:36:47.485 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.9 primary-affinity to 1 (8655362) 2023-10-20T03:36:47.510 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2023-10-20T03:36:48.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~29s 2023-10-20T03:36:48.768 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.13 to 1 (10000) 2023-10-20T03:36:48.792 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2023-10-20T03:36:49.771 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.13 primary-affinity to 1 (8655362) 2023-10-20T03:36:49.795 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-10-20T03:36:50.784 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.2 to 1 (10000) 2023-10-20T03:36:50.809 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2023-10-20T03:36:51.797 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.2 primary-affinity to 1 (8655362) 2023-10-20T03:36:51.825 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2023-10-20T03:36:52.810 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.10 to 1 (10000) 2023-10-20T03:36:52.836 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2023-10-20T03:36:53.822 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.10 primary-affinity to 1 (8655362) 2023-10-20T03:36:53.846 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2023-10-20T03:36:54.834 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.14 to 1 (10000) 2023-10-20T03:36:54.860 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2023-10-20T03:36:55.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~36s 2023-10-20T03:36:55.845 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.14 primary-affinity to 1 (8655362) 2023-10-20T03:36:55.874 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-10-20T03:36:56.857 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.3 to 1 (10000) 2023-10-20T03:36:56.881 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2023-10-20T03:36:57.868 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.3 primary-affinity to 1 (8655362) 2023-10-20T03:36:57.897 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2023-10-20T03:36:58.881 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.7 to 1 (10000) 2023-10-20T03:36:58.906 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2023-10-20T03:36:59.894 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.7 primary-affinity to 1 (8655362) 2023-10-20T03:36:59.920 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2023-10-20T03:37:00.914 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.11 to 1 (10000) 2023-10-20T03:37:00.943 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2023-10-20T03:37:01.925 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.11 primary-affinity to 1 (8655362) 2023-10-20T03:37:01.949 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2023-10-20T03:37:02.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~43s 2023-10-20T03:37:02.928 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.15 to 1 (10000) 2023-10-20T03:37:02.953 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2023-10-20T03:37:03.941 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.15 primary-affinity to 1 (8655362) 2023-10-20T03:37:03.971 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2023-10-20T03:37:04.953 INFO:teuthology.orchestra.run.smithi033.stderr:reweighted osd.6 to 1 (10000) 2023-10-20T03:37:04.980 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2023-10-20T03:37:05.965 INFO:teuthology.orchestra.run.smithi033.stderr:set osd.6 primary-affinity to 1 (8655362) 2023-10-20T03:37:05.992 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2023-10-20T03:37:05.992 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2023-10-20T03:37:05.992 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2023-10-20T03:37:05.993 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2023-10-20T03:37:05.993 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-20T03:37:05.993 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:06.293 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:06.294 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":140,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:05.960803+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:24.064001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.884878+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.632626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:21.281410+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:03.397982+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:03.463084+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.320469+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.321913+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:02.082859+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:03.991453+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:05.214755+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:25.210154+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:22.189560+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:09.314 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:09.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~50s 2023-10-20T03:37:09.620 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:09.620 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":141,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:08.772819+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:24.064001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.884878+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.632626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:21.281410+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:03.397982+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:03.463084+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.320469+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.321913+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:02.082859+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:05.214755+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:25.210154+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:22.189560+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:12.639 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:12.946 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:12.947 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":142,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:10.785606+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:24.064001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.884878+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.632626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:21.281410+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:03.397982+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.320469+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.321913+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:02.082859+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:05.214755+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:25.210154+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:22.189560+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:15.965 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:16.282 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:16.282 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":142,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:10.785606+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:24.064001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.884878+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.632626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:21.281410+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:03.397982+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.320469+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.321913+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:02.082859+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:05.214755+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:25.210154+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:22.189560+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:16.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~57s 2023-10-20T03:37:19.302 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:19.611 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:19.612 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":143,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:18.775755+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:24.064001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.884878+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:21.281410+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:03.397982+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.320469+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.321913+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:02.082859+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:25.210154+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:22.189560+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:22.632 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:22.943 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:22.943 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":144,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:22.843886+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:24.064001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.884878+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:21.281410+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.320469+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.321913+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:02.082859+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:25.210154+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:22.189560+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:23.356 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~64s 2023-10-20T03:37:25.961 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:26.270 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:26.271 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":146,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:24.865192+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:24.064001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.884878+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:21.281410+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:10.321913+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:02.082859+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:25.210154+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:29.290 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:29.600 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:29.601 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":147,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:28.892508+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:24.064001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.884878+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:21.281410+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:02.082859+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:25.210154+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:30.373 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~71s 2023-10-20T03:37:32.620 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:32.945 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:32.946 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":148,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:29.901825+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:24.064001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.884878+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:21.281410+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:25.210154+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:35.964 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:36.274 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":149,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:33.930916+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:24.064001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.884878+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:21.281410+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:37.391 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~78s 2023-10-20T03:37:39.293 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:39.600 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:39.600 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":149,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:33.930916+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:24.064001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:11.884878+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:21.281410+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:42.620 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:42.933 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:42.934 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":151,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:40.988308+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:36:24.064001+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:44.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~85s 2023-10-20T03:37:45.954 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:46.275 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:46.276 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":152,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:44.009938+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:49.293 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:49.603 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:49.603 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":152,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:44.009938+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:51.424 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~92s 2023-10-20T03:37:52.624 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:52.935 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:52.935 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":152,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:44.009938+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:55.954 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:56.262 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:56.262 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":152,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:44.009938+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:37:58.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~99s 2023-10-20T03:37:59.281 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:37:59.594 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:37:59.594 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":152,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:44.009938+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:02.614 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:02.926 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":152,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:44.009938+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:05.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~106s 2023-10-20T03:38:05.945 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:06.253 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:06.254 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":152,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:44.009938+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:09.274 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:09.581 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:09.581 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":152,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:44.009938+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:12.479 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~113s 2023-10-20T03:38:12.600 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:12.910 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:12.911 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":152,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:37:44.009938+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:02.432111+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:15.930 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:16.239 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:16.239 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":153,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:13.882412+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:19.258 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:19.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~120s 2023-10-20T03:38:19.564 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:19.564 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":153,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:13.882412+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:22.582 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:22.894 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:22.895 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":153,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:13.882412+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:25.915 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:26.222 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:26.222 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":153,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:13.882412+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:24.395027+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:05.290996+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:26.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~127s 2023-10-20T03:38:29.242 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:29.550 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:29.551 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":154,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:27.817583+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:28.317213+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:07.858465+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:17.785530+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:32.571 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:32.882 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:32.883 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":155,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:30.843235+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.760920+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:33.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~134s 2023-10-20T03:38:35.901 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:36.213 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":156,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:33.860452+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:10.375149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:39.233 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:39.543 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:39.544 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":157,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:37.896948+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:18.731227+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:22.956545+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:40.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~141s 2023-10-20T03:38:42.565 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:42.874 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:42.875 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":159,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:41.940607+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:39.952833+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:45.894 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:46.210 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:46.210 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":160,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:45.971760+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:40.251654+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:47.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~148s 2023-10-20T03:38:49.230 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:49.540 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:49.541 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":161,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:48.807601+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:43.299899+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:29.869511+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:52.560 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:52.872 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:52.873 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":162,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:51.830277+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:54.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~155s 2023-10-20T03:38:55.892 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:56.201 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:56.201 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":162,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:51.830277+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:38:59.221 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:38:59.533 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:38:59.533 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":162,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:51.830277+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:01.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~162s 2023-10-20T03:39:02.553 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:02.864 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:02.864 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":162,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:38:51.830277+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:37:33.576194+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:05.884 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:06.194 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:06.194 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":163,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:02.887643+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:08.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~169s 2023-10-20T03:39:09.214 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:09.523 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:09.524 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":163,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:02.887643+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:12.543 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:12.853 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:12.854 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":163,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:02.887643+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:15.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~176s 2023-10-20T03:39:15.874 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:16.184 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:16.184 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":163,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:02.887643+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:13.344364+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:19.202 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:19.514 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:19.514 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":164,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:16.825969+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:22.534 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:22.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~183s 2023-10-20T03:39:22.843 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:22.843 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":164,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:16.825969+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:25.861 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:26.169 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:26.169 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":164,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:16.825969+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:29.189 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:29.498 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:29.499 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":164,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:16.825969+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:29.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~190s 2023-10-20T03:39:32.518 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:32.829 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:32.830 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":164,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:16.825969+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.040206+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:27.579703+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:35.848 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:36.159 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":166,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:34.827817+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:36.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~197s 2023-10-20T03:39:39.178 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:39.489 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:39.489 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":166,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:34.827817+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.557119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:42.509 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:42.818 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:42.819 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":167,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:40.860270+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:39.594069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:33.199014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:30.737889+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:43.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~204s 2023-10-20T03:39:45.838 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:46.148 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:46.148 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":169,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:45.894123+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:26.965960+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:49.168 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:49.486 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:49.487 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":170,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:48.927000+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:37.166339+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:50.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~211s 2023-10-20T03:39:52.507 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:52.817 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:52.817 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":171,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:51.944007+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:55.837 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:56.153 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:56.153 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":171,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:51.944007+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:41.165628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:39:57.718 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~218s 2023-10-20T03:39:59.173 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:39:59.482 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:39:59.482 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":172,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:39:58.987985+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:45.813520+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:02.501 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:02.812 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:02.812 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":173,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:01.001147+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:02.715228+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:04.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~225s 2023-10-20T03:40:05.830 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:06.137 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:06.138 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":174,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:04.747280+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:09.157 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:09.462 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:09.462 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":174,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:04.747280+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:11.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~232s 2023-10-20T03:40:12.481 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:12.789 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:12.789 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":174,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:04.747280+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:47.995104+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:15.809 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:16.115 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:16.116 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":175,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:13.857346+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.147717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:18.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~239s 2023-10-20T03:40:19.134 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:19.447 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:19.447 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":176,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:18.895688+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:38:51.034512+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:22.467 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:22.777 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:22.778 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":177,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:19.905385+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:25.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~246s 2023-10-20T03:40:25.796 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:26.111 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:26.111 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":177,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:19.905385+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:29.131 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:29.442 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:29.443 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":177,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:19.905385+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:32.463 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:32.777 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:32.777 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":177,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:19.905385+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:32.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~253s 2023-10-20T03:40:35.797 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:36.108 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":177,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:19.905385+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:16.073161+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:39.129 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:39.434 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:39.435 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":178,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:36.992162+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:39.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~260s 2023-10-20T03:40:42.454 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:42.761 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:42.761 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":178,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:36.992162+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:45.780 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:46.088 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:46.088 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":178,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:36.992162+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:32.572505+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:46.827 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~267s 2023-10-20T03:40:49.105 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:49.415 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:49.416 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":179,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:48.872652+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.637048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:40.681352+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:34.674068+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:48.738121+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:52.435 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:52.746 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:52.747 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":180,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:50.884736+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.548057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.314162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.204963+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:48.738121+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:53.842 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~274s 2023-10-20T03:40:55.766 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:56.085 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:56.085 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":180,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:50.884736+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.548057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.314162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.204963+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:48.738121+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:40:59.104 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:40:59.413 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:40:59.413 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":180,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:40:50.884736+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.548057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:48.869915+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.314162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.204963+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:48.738121+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:41:00.859 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~281s 2023-10-20T03:41:02.431 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:41:02.743 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:41:02.743 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":181,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:41:00.926354+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.548057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:00.222575+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.314162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.204963+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:48.738121+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:58.119602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:41:05.763 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:41:06.079 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:41:06.079 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":182,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:41:05.953919+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.548057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:00.222575+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.314162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.204963+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:44.983226+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:48.738121+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:05.133366+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:41:07.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~288s 2023-10-20T03:41:09.097 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:41:09.412 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:41:09.412 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":183,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:41:06.964426+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.548057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:45.524625+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:00.222575+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.314162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.204963+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:05.964633+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:48.738121+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:05.133366+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:41:12.432 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:41:12.744 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:41:12.745 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":184,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:41:11.987149+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.548057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:11.479381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:00.222575+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.314162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.204963+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:05.964633+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:48.738121+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:05.133366+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:41:14.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~295s 2023-10-20T03:41:15.764 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:41:16.072 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:41:16.072 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":184,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:41:11.987149+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.548057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:13.376032+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:11.479381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:00.222575+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.314162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.204963+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:05.964633+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:48.738121+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:05.133366+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:41:19.090 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:41:19.401 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:41:19.401 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":185,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:41:17.125998+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.548057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:17.069685+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:11.479381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:39:50.970161+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:00.222575+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.314162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.204963+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:05.964633+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:48.738121+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:05.133366+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:41:21.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~302s 2023-10-20T03:41:21.906 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-10-20T03:41:22.423 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:41:22.732 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-10-20T03:41:22.733 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":186,"fsid":"b920f165-e9e7-4c3e-8d7b-cfdc002ae589","created":"2023-10-20T03:34:47.883622+0000","modified":"2023-10-20T03:41:21.150041+0000","last_up_change":"2023-10-20T03:36:16.142622+0000","last_in_change":"2023-10-20T03:36:33.309613+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-20T03:35:00.991869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-20T03:35:03.136216+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4872451c-1bc7-4f21-a95f-986fa428ea65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6825","nonce":121544157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6827","nonce":121544157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6831","nonce":121544157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":121544157},{"type":"v1","addr":"172.21.15.33:6829","nonce":121544157}]},"public_addr":"172.21.15.33:6825/121544157","cluster_addr":"172.21.15.33:6827/121544157","heartbeat_back_addr":"172.21.15.33:6831/121544157","heartbeat_front_addr":"172.21.15.33:6829/121544157","state":["exists","up"]},{"osd":1,"uuid":"0b570426-e165-4caa-a914-e1bed7d805fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6809","nonce":562043184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6811","nonce":562043184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6820","nonce":562043184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":562043184},{"type":"v1","addr":"172.21.15.40:6815","nonce":562043184}]},"public_addr":"172.21.15.40:6809/562043184","cluster_addr":"172.21.15.40:6811/562043184","heartbeat_back_addr":"172.21.15.40:6820/562043184","heartbeat_front_addr":"172.21.15.40:6815/562043184","state":["exists","up"]},{"osd":2,"uuid":"e0184c10-844b-4bcb-adaf-a0d407b708ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6825","nonce":3785405088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6827","nonce":3785405088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6831","nonce":3785405088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":3785405088},{"type":"v1","addr":"172.21.15.64:6829","nonce":3785405088}]},"public_addr":"172.21.15.64:6825/3785405088","cluster_addr":"172.21.15.64:6827/3785405088","heartbeat_back_addr":"172.21.15.64:6831/3785405088","heartbeat_front_addr":"172.21.15.64:6829/3785405088","state":["exists","up"]},{"osd":3,"uuid":"8120023b-1d48-411d-a642-2529a697b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6820","nonce":1426674535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6821","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6823","nonce":1426674535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6829","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6831","nonce":1426674535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6825","nonce":1426674535},{"type":"v1","addr":"172.21.15.170:6827","nonce":1426674535}]},"public_addr":"172.21.15.170:6820/1426674535","cluster_addr":"172.21.15.170:6823/1426674535","heartbeat_back_addr":"172.21.15.170:6831/1426674535","heartbeat_front_addr":"172.21.15.170:6827/1426674535","state":["exists","up"]},{"osd":4,"uuid":"549b3b39-41d5-4d9b-932e-a7b3e8cc0684","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6809","nonce":3186493546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6811","nonce":3186493546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6815","nonce":3186493546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":3186493546},{"type":"v1","addr":"172.21.15.33:6813","nonce":3186493546}]},"public_addr":"172.21.15.33:6809/3186493546","cluster_addr":"172.21.15.33:6811/3186493546","heartbeat_back_addr":"172.21.15.33:6815/3186493546","heartbeat_front_addr":"172.21.15.33:6813/3186493546","state":["exists","up"]},{"osd":5,"uuid":"a99ea3db-6433-4312-83d1-64ba69f028a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6817","nonce":1211670976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6823","nonce":1211670976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6829","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6831","nonce":1211670976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":1211670976},{"type":"v1","addr":"172.21.15.40:6827","nonce":1211670976}]},"public_addr":"172.21.15.40:6817/1211670976","cluster_addr":"172.21.15.40:6823/1211670976","heartbeat_back_addr":"172.21.15.40:6831/1211670976","heartbeat_front_addr":"172.21.15.40:6827/1211670976","state":["exists","up"]},{"osd":6,"uuid":"d2b38063-49eb-4f56-bb8d-411419b71b9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":28,"up_from":89,"up_thru":92,"down_at":29,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6809","nonce":2043848066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6811","nonce":2043848066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6815","nonce":2043848066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2043848066},{"type":"v1","addr":"172.21.15.64:6813","nonce":2043848066}]},"public_addr":"172.21.15.64:6809/2043848066","cluster_addr":"172.21.15.64:6811/2043848066","heartbeat_back_addr":"172.21.15.64:6815/2043848066","heartbeat_front_addr":"172.21.15.64:6813/2043848066","state":["exists","up"]},{"osd":7,"uuid":"80e34f64-35dc-4a6d-9f7f-c2263c78350d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6811","nonce":2804292733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6813","nonce":2804292733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6817","nonce":2804292733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2804292733},{"type":"v1","addr":"172.21.15.170:6815","nonce":2804292733}]},"public_addr":"172.21.15.170:6811/2804292733","cluster_addr":"172.21.15.170:6813/2804292733","heartbeat_back_addr":"172.21.15.170:6817/2804292733","heartbeat_front_addr":"172.21.15.170:6815/2804292733","state":["exists","up"]},{"osd":8,"uuid":"1cd10625-66a8-4488-b1e5-378cb0062107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6801","nonce":1638010589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6803","nonce":1638010589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6807","nonce":1638010589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1638010589},{"type":"v1","addr":"172.21.15.33:6805","nonce":1638010589}]},"public_addr":"172.21.15.33:6801/1638010589","cluster_addr":"172.21.15.33:6803/1638010589","heartbeat_back_addr":"172.21.15.33:6807/1638010589","heartbeat_front_addr":"172.21.15.33:6805/1638010589","state":["exists","up"]},{"osd":9,"uuid":"b5c70441-7f06-4bf0-b9d3-3d5c76cbd4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6801","nonce":2173797306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6803","nonce":2173797306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6807","nonce":2173797306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2173797306},{"type":"v1","addr":"172.21.15.40:6805","nonce":2173797306}]},"public_addr":"172.21.15.40:6801/2173797306","cluster_addr":"172.21.15.40:6803/2173797306","heartbeat_back_addr":"172.21.15.40:6807/2173797306","heartbeat_front_addr":"172.21.15.40:6805/2173797306","state":["exists"]},{"osd":10,"uuid":"42a67bcc-5c07-499d-8902-08c2f15e7fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6801","nonce":91701057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6803","nonce":91701057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6807","nonce":91701057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":91701057},{"type":"v1","addr":"172.21.15.64:6805","nonce":91701057}]},"public_addr":"172.21.15.64:6801/91701057","cluster_addr":"172.21.15.64:6803/91701057","heartbeat_back_addr":"172.21.15.64:6807/91701057","heartbeat_front_addr":"172.21.15.64:6805/91701057","state":["exists","up"]},{"osd":11,"uuid":"ab1db647-aada-4deb-abdc-0a1baff548db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6803","nonce":1717984751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6805","nonce":1717984751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6809","nonce":1717984751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1717984751},{"type":"v1","addr":"172.21.15.170:6807","nonce":1717984751}]},"public_addr":"172.21.15.170:6803/1717984751","cluster_addr":"172.21.15.170:6805/1717984751","heartbeat_back_addr":"172.21.15.170:6809/1717984751","heartbeat_front_addr":"172.21.15.170:6807/1717984751","state":["exists","up"]},{"osd":12,"uuid":"b20cee74-2cce-4153-97c2-56e9db40c712","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6817","nonce":3797427065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6819","nonce":3797427065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6823","nonce":3797427065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":3797427065},{"type":"v1","addr":"172.21.15.33:6821","nonce":3797427065}]},"public_addr":"172.21.15.33:6817/3797427065","cluster_addr":"172.21.15.33:6819/3797427065","heartbeat_back_addr":"172.21.15.33:6823/3797427065","heartbeat_front_addr":"172.21.15.33:6821/3797427065","state":["exists","up"]},{"osd":13,"uuid":"0974cd04-c690-4cba-ac05-5d404493dd1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6816","nonce":3041424233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6822","nonce":3041424233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6830","nonce":3041424233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":3041424233},{"type":"v1","addr":"172.21.15.40:6826","nonce":3041424233}]},"public_addr":"172.21.15.40:6816/3041424233","cluster_addr":"172.21.15.40:6822/3041424233","heartbeat_back_addr":"172.21.15.40:6830/3041424233","heartbeat_front_addr":"172.21.15.40:6826/3041424233","state":["exists","up"]},{"osd":14,"uuid":"3bf53124-13b6-4e5c-9868-2382719ffe25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6817","nonce":2940007286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6819","nonce":2940007286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6823","nonce":2940007286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2940007286},{"type":"v1","addr":"172.21.15.64:6821","nonce":2940007286}]},"public_addr":"172.21.15.64:6817/2940007286","cluster_addr":"172.21.15.64:6819/2940007286","heartbeat_back_addr":"172.21.15.64:6823/2940007286","heartbeat_front_addr":"172.21.15.64:6821/2940007286","state":["exists","up"]},{"osd":15,"uuid":"b0b5ffe5-a25b-412f-818f-b60907c0089b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6819","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6822","nonce":826332016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6826","nonce":826332016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6833","nonce":826332016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":826332016},{"type":"v1","addr":"172.21.15.170:6830","nonce":826332016}]},"public_addr":"172.21.15.170:6822/826332016","cluster_addr":"172.21.15.170:6826/826332016","heartbeat_back_addr":"172.21.15.170:6833/826332016","heartbeat_front_addr":"172.21.15.170:6830/826332016","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:19.412356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:00.605221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.548057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:17.069685+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:11.479381+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:20.225763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:00.222575+0000","dead_epoch":29},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:36.739070+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.314162+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:35:59.745876+0000","dead_epoch":94},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:18.302082+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:50.204963+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:05.964633+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:48.738121+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:40:04.695871+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-20T03:41:05.133366+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":3},{"begin":6,"length":2},{"begin":9,"length":5},{"begin":15,"length":1},{"begin":22,"length":1},{"begin":25,"length":1},{"begin":28,"length":3},{"begin":33,"length":1},{"begin":35,"length":1},{"begin":38,"length":3},{"begin":43,"length":1},{"begin":46,"length":1},{"begin":48,"length":2},{"begin":51,"length":1},{"begin":54,"length":4},{"begin":59,"length":1},{"begin":63,"length":1},{"begin":66,"length":2},{"begin":69,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-20T03:41:23.920 INFO:tasks.ceph.osd.0:Sent signal 15 2023-10-20T03:41:23.921 INFO:tasks.ceph.osd.4:Sent signal 15 2023-10-20T03:41:23.921 INFO:tasks.ceph.osd.8:Sent signal 15 2023-10-20T03:41:23.921 INFO:tasks.ceph.osd.12:Sent signal 15 2023-10-20T03:41:23.922 INFO:tasks.ceph.osd.1:Sent signal 15 2023-10-20T03:41:23.922 INFO:tasks.ceph.osd.5:Sent signal 15 2023-10-20T03:41:23.922 INFO:tasks.ceph.osd.13:Sent signal 15 2023-10-20T03:41:23.923 INFO:tasks.ceph.osd.2:Sent signal 15 2023-10-20T03:41:23.923 INFO:tasks.ceph.osd.6:Sent signal 15 2023-10-20T03:41:23.923 INFO:tasks.ceph.osd.10:Sent signal 15 2023-10-20T03:41:23.924 INFO:tasks.ceph.osd.14:Sent signal 15 2023-10-20T03:41:23.924 INFO:tasks.ceph.osd.3:Sent signal 15 2023-10-20T03:41:23.924 INFO:tasks.ceph.osd.7:Sent signal 15 2023-10-20T03:41:23.925 INFO:tasks.ceph.osd.11:Sent signal 15 2023-10-20T03:41:23.925 INFO:tasks.ceph.osd.15:Sent signal 15 2023-10-20T03:41:23.925 INFO:tasks.ceph.mon.a:Sent signal 15 2023-10-20T03:41:23.926 INFO:tasks.ceph.mon.b:Sent signal 15 2023-10-20T03:41:23.926 INFO:tasks.ceph.mon.c:Sent signal 15 2023-10-20T03:41:23.926 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-10-20T03:41:23.927 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-10-20T03:41:23.928 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:41:23.923+0000 7f3068be6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35927) UID: 0 2023-10-20T03:41:23.928 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:41:23.923+0000 7f3068be6640 -1 osd.3 186 *** Got signal Terminated *** 2023-10-20T03:41:23.928 INFO:tasks.ceph.osd.3.smithi170.stderr:2023-10-20T03:41:23.923+0000 7f3068be6640 -1 osd.3 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.929 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:41:23.922+0000 7f80e1211640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35636) UID: 0 2023-10-20T03:41:23.929 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:41:23.922+0000 7f80e1211640 -1 osd.2 186 *** Got signal Terminated *** 2023-10-20T03:41:23.929 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-10-20T03:41:23.922+0000 7f80e1211640 -1 osd.2 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.930 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:41:23.920+0000 7f8061d5f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35688) UID: 0 2023-10-20T03:41:23.930 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:41:23.920+0000 7f8061d5f640 -1 osd.0 186 *** Got signal Terminated *** 2023-10-20T03:41:23.930 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-10-20T03:41:23.920+0000 7f8061d5f640 -1 osd.0 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.930 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:41:23.923+0000 7fe4ce7d3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35929) UID: 0 2023-10-20T03:41:23.931 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:41:23.923+0000 7fe4ce7d3640 -1 osd.7 186 *** Got signal Terminated *** 2023-10-20T03:41:23.931 INFO:tasks.ceph.osd.7.smithi170.stderr:2023-10-20T03:41:23.923+0000 7fe4ce7d3640 -1 osd.7 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.931 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:41:23.922+0000 7f7212305640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 42156) UID: 0 2023-10-20T03:41:23.931 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:41:23.922+0000 7f7212305640 -1 osd.6 186 *** Got signal Terminated *** 2023-10-20T03:41:23.932 INFO:tasks.ceph.osd.6.smithi064.stderr:2023-10-20T03:41:23.922+0000 7f7212305640 -1 osd.6 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.932 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:41:23.921+0000 7f106fd35640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35543) UID: 0 2023-10-20T03:41:23.932 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:41:23.921+0000 7f106fd35640 -1 osd.1 186 *** Got signal Terminated *** 2023-10-20T03:41:23.932 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-10-20T03:41:23.921+0000 7f106fd35640 -1 osd.1 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.933 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:41:23.927+0000 7f5419c85640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35932) UID: 0 2023-10-20T03:41:23.933 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:41:23.927+0000 7f5419c85640 -1 osd.11 186 *** Got signal Terminated *** 2023-10-20T03:41:23.933 INFO:tasks.ceph.osd.11.smithi170.stderr:2023-10-20T03:41:23.927+0000 7f5419c85640 -1 osd.11 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.934 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:41:23.921+0000 7f47dadc5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35546) UID: 0 2023-10-20T03:41:23.934 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:41:23.921+0000 7f47dadc5640 -1 osd.5 186 *** Got signal Terminated *** 2023-10-20T03:41:23.934 INFO:tasks.ceph.osd.5.smithi040.stderr:2023-10-20T03:41:23.921+0000 7f47dadc5640 -1 osd.5 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.934 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:41:23.920+0000 7fa8564c8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35691) UID: 0 2023-10-20T03:41:23.935 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:41:23.920+0000 7fa8564c8640 -1 osd.4 186 *** Got signal Terminated *** 2023-10-20T03:41:23.935 INFO:tasks.ceph.osd.4.smithi033.stderr:2023-10-20T03:41:23.920+0000 7fa8564c8640 -1 osd.4 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.935 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:41:23.927+0000 7f9e54b37640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35934) UID: 0 2023-10-20T03:41:23.935 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:41:23.927+0000 7f9e54b37640 -1 osd.15 186 *** Got signal Terminated *** 2023-10-20T03:41:23.936 INFO:tasks.ceph.osd.15.smithi170.stderr:2023-10-20T03:41:23.927+0000 7f9e54b37640 -1 osd.15 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.936 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:41:23.922+0000 7fd15ef99640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35641) UID: 0 2023-10-20T03:41:23.936 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:41:23.922+0000 7fd15ef99640 -1 osd.10 186 *** Got signal Terminated *** 2023-10-20T03:41:23.936 INFO:tasks.ceph.osd.10.smithi064.stderr:2023-10-20T03:41:23.922+0000 7fd15ef99640 -1 osd.10 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.936 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:41:23.921+0000 7f23088c5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35550) UID: 0 2023-10-20T03:41:23.937 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:41:23.921+0000 7f23088c5640 -1 osd.13 186 *** Got signal Terminated *** 2023-10-20T03:41:23.937 INFO:tasks.ceph.osd.13.smithi040.stderr:2023-10-20T03:41:23.921+0000 7f23088c5640 -1 osd.13 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.937 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:41:23.920+0000 7f5261bac640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35693) UID: 0 2023-10-20T03:41:23.937 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:41:23.920+0000 7f5261bac640 -1 osd.8 186 *** Got signal Terminated *** 2023-10-20T03:41:23.937 INFO:tasks.ceph.osd.8.smithi033.stderr:2023-10-20T03:41:23.920+0000 7f5261bac640 -1 osd.8 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.938 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:41:23.922+0000 7f201c773640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35643) UID: 0 2023-10-20T03:41:23.938 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:41:23.922+0000 7f201c773640 -1 osd.14 186 *** Got signal Terminated *** 2023-10-20T03:41:23.938 INFO:tasks.ceph.osd.14.smithi064.stderr:2023-10-20T03:41:23.922+0000 7f201c773640 -1 osd.14 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.939 INFO:tasks.ceph.mon.b.smithi040.stderr:2023-10-20T03:41:23.925+0000 7f8df7c49640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 35491) UID: 0 2023-10-20T03:41:23.939 INFO:tasks.ceph.mon.b.smithi040.stderr:2023-10-20T03:41:23.925+0000 7f8df7c49640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-10-20T03:41:23.939 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:41:23.920+0000 7f1813d72640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35695) UID: 0 2023-10-20T03:41:23.939 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:41:23.920+0000 7f1813d72640 -1 osd.12 186 *** Got signal Terminated *** 2023-10-20T03:41:23.939 INFO:tasks.ceph.osd.12.smithi033.stderr:2023-10-20T03:41:23.920+0000 7f1813d72640 -1 osd.12 186 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-20T03:41:23.940 INFO:tasks.ceph.mon.c.smithi064.stderr:2023-10-20T03:41:23.926+0000 7fb42acab640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 35584) UID: 0 2023-10-20T03:41:23.940 INFO:tasks.ceph.mon.c.smithi064.stderr:2023-10-20T03:41:23.926+0000 7fb42acab640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-10-20T03:41:23.940 INFO:tasks.ceph.mon.a.smithi033.stderr:2023-10-20T03:41:23.924+0000 7f44dcebd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 35576) UID: 0 2023-10-20T03:41:23.940 INFO:tasks.ceph.mon.a.smithi033.stderr:2023-10-20T03:41:23.924+0000 7f44dcebd640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-20T03:41:24.128 INFO:tasks.ceph.mgr.y.smithi033.stderr:daemon-helper: command crashed with signal 15 2023-10-20T03:41:25.752 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-20T03:43:25.773 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-10-20T03:43:25.775 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-10-20T03:43:25.775 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-20T03:43:25.786 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-20T03:43:25.786 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-20T03:45:25.806 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-10-20T03:45:25.808 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-20T03:45:25.809 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-20T03:45:25.809 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-20T03:45:25.810 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-20T03:45:25.810 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.811 INFO:tasks.ceph.osd.0:Stopped 2023-10-20T03:45:25.811 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-20T03:45:25.812 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.813 INFO:tasks.ceph.osd.4:Stopped 2023-10-20T03:45:25.813 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2023-10-20T03:45:25.814 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.814 INFO:tasks.ceph.osd.8:Stopped 2023-10-20T03:45:25.815 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2023-10-20T03:45:25.815 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.816 INFO:tasks.ceph.osd.12:Stopped 2023-10-20T03:45:25.816 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-20T03:45:25.817 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.818 INFO:tasks.ceph.osd.1:Stopped 2023-10-20T03:45:25.818 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-20T03:45:25.819 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.819 INFO:tasks.ceph.osd.5:Stopped 2023-10-20T03:45:25.820 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2023-10-20T03:45:25.820 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:45:25.821 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9' 2023-10-20T03:45:25.822 INFO:tasks.ceph.osd.9:Stopped 2023-10-20T03:45:25.823 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2023-10-20T03:45:25.823 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.824 INFO:tasks.ceph.osd.13:Stopped 2023-10-20T03:45:25.825 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-20T03:45:25.826 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.827 INFO:tasks.ceph.osd.2:Stopped 2023-10-20T03:45:25.827 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-20T03:45:25.828 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.829 INFO:tasks.ceph.osd.6:Stopped 2023-10-20T03:45:25.829 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2023-10-20T03:45:25.830 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.830 INFO:tasks.ceph.osd.10:Stopped 2023-10-20T03:45:25.831 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2023-10-20T03:45:25.831 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.832 INFO:tasks.ceph.osd.14:Stopped 2023-10-20T03:45:25.832 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-20T03:45:25.833 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.834 INFO:tasks.ceph.osd.3:Stopped 2023-10-20T03:45:25.834 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-20T03:45:25.835 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.835 INFO:tasks.ceph.osd.7:Stopped 2023-10-20T03:45:25.836 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2023-10-20T03:45:25.836 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.837 INFO:tasks.ceph.osd.11:Stopped 2023-10-20T03:45:25.838 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2023-10-20T03:45:25.838 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.839 INFO:tasks.ceph.osd.15:Stopped 2023-10-20T03:45:25.839 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-20T03:45:25.840 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-20T03:45:25.840 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:45:25.841 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y' 2023-10-20T03:45:25.842 INFO:tasks.ceph.mgr.y:Stopped 2023-10-20T03:45:25.843 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-20T03:45:25.843 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.844 INFO:tasks.ceph.mgr.x:Stopped 2023-10-20T03:45:25.845 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-20T03:45:25.845 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-20T03:45:25.846 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.846 INFO:tasks.ceph.mon.a:Stopped 2023-10-20T03:45:25.847 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-20T03:45:25.847 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.848 INFO:tasks.ceph.mon.b:Stopped 2023-10-20T03:45:25.849 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-20T03:45:25.849 INFO:teuthology.orchestra.run:waiting for 300 2023-10-20T03:45:25.850 INFO:tasks.ceph.mon.c:Stopped 2023-10-20T03:45:25.850 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-20T03:45:25.851 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | head -n 1 2023-10-20T03:45:25.868 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-20T03:45:25.869 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-20T03:45:26.047 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-20T03:45:26.048 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-20T03:45:26.202 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-20T03:45:26.203 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2023-10-20T03:45:26.317 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi033.front.sepia.ceph.com 2023-10-20T03:45:26.318 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2023-10-20T03:45:26.441 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-20T03:45:26.442 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-20T03:45:26.515 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-20T03:45:26.516 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-20T03:45:26.668 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-20T03:45:26.669 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2023-10-20T03:45:26.826 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-20T03:45:26.826 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2023-10-20T03:45:26.988 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-20T03:45:26.989 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-20T03:45:27.040 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-20T03:45:27.041 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-20T03:45:27.333 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-20T03:45:27.334 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2023-10-20T03:45:27.474 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi064.front.sepia.ceph.com 2023-10-20T03:45:27.475 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2023-10-20T03:45:27.634 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-20T03:45:27.635 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-20T03:45:27.698 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-20T03:45:27.699 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-20T03:45:27.873 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-20T03:45:27.873 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2023-10-20T03:45:28.055 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi170.front.sepia.ceph.com 2023-10-20T03:45:28.056 DEBUG:teuthology.orchestra.run.smithi170:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2023-10-20T03:45:28.214 INFO:tasks.ceph:Archiving mon data... 2023-10-20T03:45:28.216 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/data/mon.a.tgz 2023-10-20T03:45:28.216 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2023-10-20T03:45:28.223 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.L4nV8iVeVt 2023-10-20T03:45:28.223 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.L4nV8iVeVt 2023-10-20T03:45:28.409 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.L4nV8iVeVt 2023-10-20T03:45:28.477 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.L4nV8iVeVt is 558KB 2023-10-20T03:45:28.537 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.L4nV8iVeVt 2023-10-20T03:45:28.543 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/data/mon.b.tgz 2023-10-20T03:45:28.544 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2023-10-20T03:45:28.550 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.Rompkbl5RZ 2023-10-20T03:45:28.551 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Rompkbl5RZ 2023-10-20T03:45:28.776 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.Rompkbl5RZ 2023-10-20T03:45:28.836 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.Rompkbl5RZ is 769KB 2023-10-20T03:45:28.904 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.Rompkbl5RZ 2023-10-20T03:45:28.910 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/data/mon.c.tgz 2023-10-20T03:45:28.910 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2023-10-20T03:45:28.917 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.o0x5qoMS14 2023-10-20T03:45:28.917 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.o0x5qoMS14 2023-10-20T03:45:29.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.o0x5qoMS14 2023-10-20T03:45:29.212 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.o0x5qoMS14 is 768KB 2023-10-20T03:45:29.283 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.o0x5qoMS14 2023-10-20T03:45:29.289 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-20T03:45:29.289 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-20T03:45:29.292 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-20T03:45:29.295 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-20T03:45:29.334 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-20T03:45:29.349 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-20T03:45:29.350 DEBUG:teuthology.orchestra.run.smithi033:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-20T03:45:29.352 DEBUG:teuthology.orchestra.run.smithi040:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-20T03:45:29.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-20T03:45:29.371 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-20T03:45:29.383 INFO:teuthology.orchestra.run.smithi040.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-20T03:45:29.390 DEBUG:teuthology.orchestra.run.smithi170:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-20T03:45:29.405 INFO:teuthology.orchestra.run.smithi064.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-20T03:45:29.648 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-20T03:45:29.652 INFO:tasks.ceph:Archiving crash dumps... 2023-10-20T03:45:29.654 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/remote/smithi033/crash 2023-10-20T03:45:29.656 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-20T03:45:29.671 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/remote/smithi040/crash 2023-10-20T03:45:29.673 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-20T03:45:29.728 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/remote/smithi064/crash 2023-10-20T03:45:29.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-20T03:45:29.745 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/remote/smithi170/crash 2023-10-20T03:45:29.746 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-20T03:45:29.760 INFO:tasks.ceph:Compressing logs... 2023-10-20T03:45:29.760 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-20T03:45:29.764 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-20T03:45:29.766 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-20T03:45:29.775 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44537.log 2023-10-20T03:45:29.776 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44979.log 2023-10-20T03:45:29.776 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44537.log.gz 2023-10-20T03:45:29.777 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40905.log 2023-10-20T03:45:29.779 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44142.log 2023-10-20T03:45:29.779 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44979.log.gz 2023-10-20T03:45:29.779 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43435.log 2023-10-20T03:45:29.779 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40905.log.gz 2023-10-20T03:45:29.779 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45135.log 2023-10-20T03:45:29.780 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44142.log.gz 2023-10-20T03:45:29.780 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43435.log.gz 2023-10-20T03:45:29.780 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44953.log 2023-10-20T03:45:29.780 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39253.log 2023-10-20T03:45:29.781 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45135.log.gz 2023-10-20T03:45:29.781 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.44953.log: /var/log/ceph/ceph-client.admin.45005.log 2023-10-20T03:45:29.781 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44953.log.gz 2023-10-20T03:45:29.785 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-20T03:45:29.786 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-20T03:45:29.786 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-20T03:45:29.786 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-20T03:45:29.786 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2023-10-20T03:45:29.787 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2023-10-20T03:45:29.787 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-20T03:45:29.787 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.13.log: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-20T03:45:29.787 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mon.b.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-20T03:45:29.797 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-20T03:45:29.813 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39253.log.gz 2023-10-20T03:45:29.813 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-20T03:45:29.813 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41596.log 2023-10-20T03:45:29.814 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45005.log.gz 2023-10-20T03:45:29.814 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43700.log 2023-10-20T03:45:29.814 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.41596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39609.log 2023-10-20T03:45:29.814 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41596.log.gz 2023-10-20T03:45:29.814 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39244.log 2023-10-20T03:45:29.815 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43700.log.gz 2023-10-20T03:45:29.815 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39609.log.gz 2023-10-20T03:45:29.815 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45941.log 2023-10-20T03:45:29.815 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39642.log 2023-10-20T03:45:29.815 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39244.log.gz 2023-10-20T03:45:29.816 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45603.log 2023-10-20T03:45:29.816 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45941.log.gz 2023-10-20T03:45:29.816 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39642.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-20T03:45:29.816 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39642.log.gz 2023-10-20T03:45:29.816 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43882.log 2023-10-20T03:45:29.817 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45603.log.gz 2023-10-20T03:45:29.817 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42260.log 2023-10-20T03:45:29.817 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.43882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44455.log 2023-10-20T03:45:29.817 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43882.log.gz 2023-10-20T03:45:29.817 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42719.log 2023-10-20T03:45:29.817 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42260.log.gz 2023-10-20T03:45:29.818 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44455.log.gz 2023-10-20T03:45:29.818 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43726.log 2023-10-20T03:45:29.818 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.42719.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40131.log 2023-10-20T03:45:29.818 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42719.log.gz 2023-10-20T03:45:29.818 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43726.log.gz 2023-10-20T03:45:29.819 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44273.log 2023-10-20T03:45:29.819 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40105.log 2023-10-20T03:45:29.819 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40131.log.gz 2023-10-20T03:45:29.819 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44273.log.gz 2023-10-20T03:45:29.819 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44012.log 2023-10-20T03:45:29.820 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44064.log 2023-10-20T03:45:29.820 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40105.log.gz 2023-10-20T03:45:29.820 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44012.log.gz 2023-10-20T03:45:29.820 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43778.log 2023-10-20T03:45:29.820 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45421.log 2023-10-20T03:45:29.820 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44064.log.gz 2023-10-20T03:45:29.821 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43778.log.gz 2023-10-20T03:45:29.821 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39672.log 2023-10-20T03:45:29.821 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39169.log 2023-10-20T03:45:29.821 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45421.log.gz 2023-10-20T03:45:29.821 INFO:teuthology.orchestra.run.smithi033.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-20T03:45:29.822 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39672.log.gz 2023-10-20T03:45:29.822 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39633.log 2023-10-20T03:45:29.822 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.39169.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40456.log 2023-10-20T03:45:29.822 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39169.log.gz 2023-10-20T03:45:29.822 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45525.log 2023-10-20T03:45:29.823 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39633.log.gz 2023-10-20T03:45:29.823 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40456.log.gz 2023-10-20T03:45:29.823 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44195.log 2023-10-20T03:45:29.823 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39651.log 2023-10-20T03:45:29.823 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45525.log.gz 2023-10-20T03:45:29.823 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45083.log 2023-10-20T03:45:29.824 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44195.log.gz 2023-10-20T03:45:29.824 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45837.log 2023-10-20T03:45:29.824 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39651.log.gz 2023-10-20T03:45:29.824 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2023-10-20T03:45:29.824 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45083.log.gz 2023-10-20T03:45:29.825 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45759.log 2023-10-20T03:45:29.825 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45837.log.gz 2023-10-20T03:45:29.825 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41385.log 2023-10-20T03:45:29.825 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43204.log 2023-10-20T03:45:29.825 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45759.log.gz 2023-10-20T03:45:29.826 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41385.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41385.log.gz 2023-10-20T03:45:29.826 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44849.log 2023-10-20T03:45:29.826 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42367.log 2023-10-20T03:45:29.826 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43204.log.gz 2023-10-20T03:45:29.826 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44849.log.gz 2023-10-20T03:45:29.826 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44116.log 2023-10-20T03:45:29.827 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39606.log 2023-10-20T03:45:29.827 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42367.log.gz 2023-10-20T03:45:29.827 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44116.log.gz 2023-10-20T03:45:29.827 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46232.log 2023-10-20T03:45:29.827 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42341.log 2023-10-20T03:45:29.828 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39606.log.gz 2023-10-20T03:45:29.828 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46232.log.gzgzip 2023-10-20T03:45:29.828 INFO:teuthology.orchestra.run.smithi033.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42981.log 2023-10-20T03:45:29.828 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44823.log 2023-10-20T03:45:29.828 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42341.log.gz 2023-10-20T03:45:29.829 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42981.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.42981.log.gz 2023-10-20T03:45:29.829 INFO:teuthology.orchestra.run.smithi033.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45499.log 2023-10-20T03:45:29.829 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46415.log 2023-10-20T03:45:29.829 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44823.log.gz 2023-10-20T03:45:29.829 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45499.log.gz 2023-10-20T03:45:29.829 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45915.log 2023-10-20T03:45:29.830 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44927.log 2023-10-20T03:45:29.830 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46415.log.gz 2023-10-20T03:45:29.830 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45915.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-mon.a.log 2023-10-20T03:45:29.830 INFO:teuthology.orchestra.run.smithi033.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45915.log.gz 2023-10-20T03:45:29.830 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39213.log 2023-10-20T03:45:29.831 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44927.log.gz 2023-10-20T03:45:29.831 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41013.log 2023-10-20T03:45:29.831 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43276.log 2023-10-20T03:45:29.831 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39213.log.gz 2023-10-20T03:45:29.831 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44377.log 2023-10-20T03:45:29.832 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41013.log.gz 2023-10-20T03:45:29.832 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41541.log 2023-10-20T03:45:29.832 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43276.log.gz 2023-10-20T03:45:29.832 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44481.log 2023-10-20T03:45:29.832 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44377.log.gz 2023-10-20T03:45:29.833 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45161.log 2023-10-20T03:45:29.833 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41541.log.gz 2023-10-20T03:45:29.833 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39252.log 2023-10-20T03:45:29.833 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44481.log.gz 2023-10-20T03:45:29.833 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41411.log 2023-10-20T03:45:29.833 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45161.log.gz 2023-10-20T03:45:29.834 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46206.log 2023-10-20T03:45:29.834 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39252.log.gz 2023-10-20T03:45:29.834 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44771.log 2023-10-20T03:45:29.834 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41411.log.gz 2023-10-20T03:45:29.834 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41647.log 2023-10-20T03:45:29.835 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46206.log.gz 2023-10-20T03:45:29.835 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.44771.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40498.log 2023-10-20T03:45:29.835 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44771.log.gz 2023-10-20T03:45:29.835 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43088.log 2023-10-20T03:45:29.835 INFO:teuthology.orchestra.run.smithi033.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.41647.log.gz 2023-10-20T03:45:29.836 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42749.log 2023-10-20T03:45:29.836 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40498.log.gz 2023-10-20T03:45:29.836 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45057.log 2023-10-20T03:45:29.836 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43088.log.gz 2023-10-20T03:45:29.836 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43155.log 2023-10-20T03:45:29.837 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42749.log.gz 2023-10-20T03:45:29.837 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45681.log 2023-10-20T03:45:29.837 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45057.log.gz 2023-10-20T03:45:29.837 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45967.log 2023-10-20T03:45:29.837 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43155.log.gz 2023-10-20T03:45:29.837 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39641.log 2023-10-20T03:45:29.838 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45681.log.gz 2023-10-20T03:45:29.838 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43908.log 2023-10-20T03:45:29.838 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45967.log.gz 2023-10-20T03:45:29.838 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42577.log 2023-10-20T03:45:29.839 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39641.log.gz 2023-10-20T03:45:29.839 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43648.log 2023-10-20T03:45:29.841 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43908.log.gz 2023-10-20T03:45:29.841 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43487.log 2023-10-20T03:45:29.841 INFO:teuthology.orchestra.run.smithi033.stderr: 23.9% -- replaced with /var/log/ceph/ceph-client.admin.42577.log.gz 2023-10-20T03:45:29.842 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40788.log 2023-10-20T03:45:29.842 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43648.log.gz 2023-10-20T03:45:29.842 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.43487.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43006.log 2023-10-20T03:45:29.843 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43487.log.gz 2023-10-20T03:45:29.843 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44299.log 2023-10-20T03:45:29.843 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40788.log.gz 2023-10-20T03:45:29.843 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39117.log 2023-10-20T03:45:29.843 INFO:teuthology.orchestra.run.smithi033.stderr: 25.0% -- replaced with /var/log/ceph/ceph-client.admin.43006.log.gz 2023-10-20T03:45:29.844 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43332.log 2023-10-20T03:45:29.844 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44299.log.gz 2023-10-20T03:45:29.844 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44403.log 2023-10-20T03:45:29.844 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39117.log.gz 2023-10-20T03:45:29.844 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40157.log 2023-10-20T03:45:29.845 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43332.log.gz 2023-10-20T03:45:29.845 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41852.log 2023-10-20T03:45:29.845 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44403.log.gz 2023-10-20T03:45:29.845 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40157.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40157.log.gz -5 2023-10-20T03:45:29.845 INFO:teuthology.orchestra.run.smithi033.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41038.log 2023-10-20T03:45:29.846 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43491.log 2023-10-20T03:45:29.846 INFO:teuthology.orchestra.run.smithi033.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.41852.log.gz 2023-10-20T03:45:29.846 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43830.log 2023-10-20T03:45:29.846 INFO:teuthology.orchestra.run.smithi033.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.41038.log.gz 2023-10-20T03:45:29.846 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43752.log 2023-10-20T03:45:29.846 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43491.log.gz 2023-10-20T03:45:29.847 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40384.log 2023-10-20T03:45:29.847 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43830.log.gz 2023-10-20T03:45:29.847 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45291.log 2023-10-20T03:45:29.847 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43752.log.gz 2023-10-20T03:45:29.847 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44719.log 2023-10-20T03:45:29.848 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40384.log.gz 2023-10-20T03:45:29.859 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.41196.log 2023-10-20T03:45:29.859 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-20T03:45:29.860 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.41196.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.38937.log 2023-10-20T03:45:29.860 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-20T03:45:29.860 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.39678.log 2023-10-20T03:45:29.860 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.38937.log: /var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2023-10-20T03:45:29.860 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.39678.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-20T03:45:29.860 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log/var/log/ceph/ceph-osd.14.log: 2023-10-20T03:45:29.861 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-20T03:45:29.861 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.c.log: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-20T03:45:29.861 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.40423.log 2023-10-20T03:45:29.861 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.audit.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-20T03:45:29.861 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2023-10-20T03:45:29.862 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.40423.log: 93.0% -- replaced with /var/log/ceph/objectstore_tool.41196.log.gz 2023-10-20T03:45:29.862 INFO:teuthology.orchestra.run.smithi064.stderr: 93.3% -- replaced with /var/log/ceph/objectstore_tool.39678.log.gz 2023-10-20T03:45:29.862 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.10.log: 93.3% -- replaced with /var/log/ceph/objectstore_tool.40423.log.gz 2023-10-20T03:45:29.862 INFO:teuthology.orchestra.run.smithi064.stderr: 93.6% -- replaced with /var/log/ceph/objectstore_tool.38937.log.gz 2023-10-20T03:45:29.869 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35839.log 2023-10-20T03:45:29.869 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35615.log 2023-10-20T03:45:29.869 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.39300.log 2023-10-20T03:45:29.870 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35839.log.gz 2023-10-20T03:45:29.870 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35694.log 2023-10-20T03:45:29.870 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35615.log.gz 2023-10-20T03:45:29.870 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.0.39300.log: 75.4% -- replaced with /var/log/ceph/ceph-client.0.39300.log.gz 2023-10-20T03:45:29.870 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35543.log 2023-10-20T03:45:29.871 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35519.log 2023-10-20T03:45:29.871 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35694.log.gz 2023-10-20T03:45:29.871 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35591.log 2023-10-20T03:45:29.871 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35543.log: /var/log/ceph/ceph-client.admin.35519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35766.log 2023-10-20T03:45:29.871 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35519.log.gz 2023-10-20T03:45:29.872 INFO:teuthology.orchestra.run.smithi170.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35543.log.gz 2023-10-20T03:45:29.872 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-20T03:45:29.872 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35591.log.gz 2023-10-20T03:45:29.872 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35766.log.gz 2023-10-20T03:45:29.872 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-20T03:45:29.872 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2023-10-20T03:45:29.873 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35814.log 2023-10-20T03:45:29.873 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35790.log 2023-10-20T03:45:29.873 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35814.log.gz 2023-10-20T03:45:29.873 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35863.log 2023-10-20T03:45:29.873 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35639.log 2023-10-20T03:45:29.874 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35790.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35790.log.gz 2023-10-20T03:45:29.874 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35863.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.35863.log.gz --verbose 2023-10-20T03:45:29.874 INFO:teuthology.orchestra.run.smithi170.stderr: -- /var/log/ceph/ceph-client.admin.35663.log 2023-10-20T03:45:29.874 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35639.log.gz 2023-10-20T03:45:29.874 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2023-10-20T03:45:29.875 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-20T03:45:29.875 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35663.log.gz 2023-10-20T03:45:29.875 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35742.log 2023-10-20T03:45:29.875 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35887.log 2023-10-20T03:45:29.875 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35742.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35742.log.gz 2023-10-20T03:45:29.875 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/ceph-client.admin.35567.log 2023-10-20T03:45:29.876 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35887.log.gz 2023-10-20T03:45:29.876 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35718.log 2023-10-20T03:45:29.876 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35567.log.gz 2023-10-20T03:45:29.876 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/ceph-client.admin.35718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35718.log.gz 2023-10-20T03:45:29.916 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43596.log 2023-10-20T03:45:29.916 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45291.log.gz 2023-10-20T03:45:29.917 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41853.log 2023-10-20T03:45:29.917 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44719.log.gz 2023-10-20T03:45:29.917 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39221.log 2023-10-20T03:45:29.917 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43596.log.gz 2023-10-20T03:45:29.917 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43960.log 2023-10-20T03:45:29.917 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41853.log.gz 2023-10-20T03:45:29.918 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39079.log 2023-10-20T03:45:29.918 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39221.log.gz 2023-10-20T03:45:29.918 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40389.log 2023-10-20T03:45:29.918 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43960.log.gz 2023-10-20T03:45:29.918 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43439.log 2023-10-20T03:45:29.919 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39079.log.gz 2023-10-20T03:45:29.919 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44351.log 2023-10-20T03:45:29.919 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40389.log.gz 2023-10-20T03:45:29.919 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45655.log 2023-10-20T03:45:29.919 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43439.log.gz 2023-10-20T03:45:29.920 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40668.log 2023-10-20T03:45:29.920 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44351.log.gz 2023-10-20T03:45:29.920 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39233.log 2023-10-20T03:45:29.920 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45655.log.gz 2023-10-20T03:45:29.920 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39250.log 2023-10-20T03:45:29.921 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40668.log.gz 2023-10-20T03:45:29.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39233.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.39647.log -- replaced with /var/log/ceph/ceph-client.admin.39233.log.gz 2023-10-20T03:45:29.921 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-10-20T03:45:29.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44090.log 2023-10-20T03:45:29.921 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39250.log.gz 2023-10-20T03:45:29.921 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42640.log 2023-10-20T03:45:29.922 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39647.log.gz 2023-10-20T03:45:29.922 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41162.log 2023-10-20T03:45:29.922 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44090.log.gz 2023-10-20T03:45:29.922 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40027.log 2023-10-20T03:45:29.922 INFO:teuthology.orchestra.run.smithi033.stderr: 23.9% -- replaced with /var/log/ceph/ceph-client.admin.42640.log.gz 2023-10-20T03:45:29.923 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46097.log 2023-10-20T03:45:29.923 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41162.log.gz 2023-10-20T03:45:29.923 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39254.log 2023-10-20T03:45:29.923 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40027.log.gz 2023-10-20T03:45:29.923 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46258.log 2023-10-20T03:45:29.923 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46097.log.gz 2023-10-20T03:45:29.924 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43376.log 2023-10-20T03:45:29.924 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39254.log.gz 2023-10-20T03:45:29.924 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45863.log 2023-10-20T03:45:29.924 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46258.log.gz 2023-10-20T03:45:29.924 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44745.log 2023-10-20T03:45:29.925 INFO:teuthology.orchestra.run.smithi033.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.43376.log.gz 2023-10-20T03:45:29.925 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35704.log 2023-10-20T03:45:29.925 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45863.log.gz 2023-10-20T03:45:29.925 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44745.log.gz 2023-10-20T03:45:29.925 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40955.log 2023-10-20T03:45:29.926 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39646.log 2023-10-20T03:45:29.926 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35704.log.gz 2023-10-20T03:45:29.926 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41703.log 2023-10-20T03:45:29.926 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40955.log.gz 2023-10-20T03:45:29.926 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45785.log 2023-10-20T03:45:29.927 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39646.log.gz 2023-10-20T03:45:29.927 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41703.log.gz 2023-10-20T03:45:29.927 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45109.log 2023-10-20T03:45:29.927 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44247.log 2023-10-20T03:45:29.927 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45785.log.gz 2023-10-20T03:45:29.928 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42929.log 2023-10-20T03:45:29.928 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45109.log.gz 2023-10-20T03:45:29.928 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45265.log 2023-10-20T03:45:29.928 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44247.log.gz 2023-10-20T03:45:29.928 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45551.log 2023-10-20T03:45:29.928 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42929.log.gz 2023-10-20T03:45:29.929 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44563.log 2023-10-20T03:45:29.929 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45265.log.gz 2023-10-20T03:45:29.929 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43856.log 2023-10-20T03:45:29.929 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45551.log.gz 2023-10-20T03:45:29.929 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44563.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44563.log.gz 2023-10-20T03:45:29.930 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45889.log 2023-10-20T03:45:29.930 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46071.log 2023-10-20T03:45:29.930 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43856.log.gz 2023-10-20T03:45:29.930 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42189.log 2023-10-20T03:45:29.930 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45889.log.gz 2023-10-20T03:45:29.931 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46284.log 2023-10-20T03:45:29.931 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46071.log.gz 2023-10-20T03:45:29.931 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42189.log.gz 2023-10-20T03:45:29.931 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44901.log 2023-10-20T03:45:29.931 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43542.log 2023-10-20T03:45:29.931 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46284.log.gz 2023-10-20T03:45:29.932 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44901.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.44901.log.gz 2023-10-20T03:45:29.932 INFO:teuthology.orchestra.run.smithi033.stderr: /var/log/ceph/ceph-client.admin.41991.log 2023-10-20T03:45:29.932 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44875.log 2023-10-20T03:45:29.932 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43542.log.gz 2023-10-20T03:45:29.932 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45811.log 2023-10-20T03:45:29.933 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41991.log.gz 2023-10-20T03:45:29.933 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41330.log 2023-10-20T03:45:29.933 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44875.log.gz 2023-10-20T03:45:29.933 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45811.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45811.log.gz 2023-10-20T03:45:29.933 INFO:teuthology.orchestra.run.smithi033.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2023-10-20T03:45:29.934 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43570.log 2023-10-20T03:45:29.934 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41330.log.gz 2023-10-20T03:45:29.934 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46430.log 2023-10-20T03:45:29.934 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45447.log 2023-10-20T03:45:29.934 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43570.log.gz 2023-10-20T03:45:29.935 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45993.log 2023-10-20T03:45:29.935 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46430.log.gz 2023-10-20T03:45:29.935 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45447.log.gz 2023-10-20T03:45:29.935 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44429.log 2023-10-20T03:45:29.935 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46154.log 2023-10-20T03:45:29.936 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45993.log.gz 2023-10-20T03:45:29.936 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39669.log 2023-10-20T03:45:29.936 INFO:teuthology.orchestra.run.smithi033.stderr: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-20T03:45:29.936 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44429.log.gz 2023-10-20T03:45:29.936 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46154.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46154.log.gz -5 2023-10-20T03:45:29.937 INFO:teuthology.orchestra.run.smithi033.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43804.log 2023-10-20T03:45:29.937 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41858.log 2023-10-20T03:45:29.937 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39669.log.gz 2023-10-20T03:45:29.937 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39222.log 2023-10-20T03:45:29.937 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43804.log.gz 2023-10-20T03:45:29.937 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44641.log 2023-10-20T03:45:29.938 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41858.log.gz 2023-10-20T03:45:29.938 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39255.log 2023-10-20T03:45:29.938 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39222.log.gz 2023-10-20T03:45:29.938 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40053.log 2023-10-20T03:45:29.938 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44641.log.gz 2023-10-20T03:45:29.939 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45395.log 2023-10-20T03:45:29.939 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39255.log.gz 2023-10-20T03:45:29.939 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40053.log: -- /var/log/ceph/ceph-client.admin.42127.log 2023-10-20T03:45:29.939 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40053.log.gz 2023-10-20T03:45:29.939 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41503.log 2023-10-20T03:45:29.940 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45395.log.gz 2023-10-20T03:45:29.940 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43674.log 2023-10-20T03:45:29.940 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42127.log.gz 2023-10-20T03:45:29.940 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5/var/log/ceph/ceph-client.admin.41503.log: --verbose -- /var/log/ceph/ceph-client.admin.46310.log 2023-10-20T03:45:29.940 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41503.log.gz 2023-10-20T03:45:29.941 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43062.log 2023-10-20T03:45:29.941 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43674.log.gz 2023-10-20T03:45:29.941 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42636.log 2023-10-20T03:45:29.941 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46310.log.gz 2023-10-20T03:45:29.941 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39248.log 2023-10-20T03:45:29.942 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43062.log.gz 2023-10-20T03:45:29.942 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44038.log 2023-10-20T03:45:29.942 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42636.log.gz 2023-10-20T03:45:29.942 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41110.log 2023-10-20T03:45:29.942 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39248.log.gz 2023-10-20T03:45:29.943 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42553.log 2023-10-20T03:45:29.943 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44038.log.gz 2023-10-20T03:45:29.943 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41964.log 2023-10-20T03:45:29.943 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41110.log.gz 2023-10-20T03:45:29.943 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40226.log 2023-10-20T03:45:29.943 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42553.log.gz 2023-10-20T03:45:29.944 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40230.log 2023-10-20T03:45:29.944 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41964.log.gz 2023-10-20T03:45:29.944 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40717.log 2023-10-20T03:45:29.944 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40226.log.gz 2023-10-20T03:45:29.944 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39041.log 2023-10-20T03:45:29.945 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40230.log.gz 2023-10-20T03:45:29.945 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42426.log 2023-10-20T03:45:29.945 INFO:teuthology.orchestra.run.smithi033.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.40717.log.gz 2023-10-20T03:45:29.945 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46389.log 2023-10-20T03:45:29.945 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39041.log.gz 2023-10-20T03:45:29.945 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40079.log 2023-10-20T03:45:29.946 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42426.log.gz 2023-10-20T03:45:29.946 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39143.log 2023-10-20T03:45:29.946 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46389.log.gz 2023-10-20T03:45:29.946 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43331.log 2023-10-20T03:45:29.946 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40079.log.gz 2023-10-20T03:45:29.947 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40187.log 2023-10-20T03:45:29.947 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39143.log.gz 2023-10-20T03:45:29.947 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39610.log 2023-10-20T03:45:29.947 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43331.log.gz 2023-10-20T03:45:29.947 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42816.log 2023-10-20T03:45:29.948 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40187.log.gz 2023-10-20T03:45:29.948 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45031.log 2023-10-20T03:45:29.948 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39610.log.gz 2023-10-20T03:45:29.948 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46336.log 2023-10-20T03:45:29.948 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42816.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42816.log.gz 2023-10-20T03:45:29.948 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35583.log 2023-10-20T03:45:29.949 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45031.log.gz 2023-10-20T03:45:29.949 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45577.log 2023-10-20T03:45:29.949 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46336.log.gz 2023-10-20T03:45:29.949 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.35583.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43986.log 2023-10-20T03:45:29.949 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35583.log.gz 2023-10-20T03:45:29.950 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39231.log 2023-10-20T03:45:29.950 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45577.log.gz 2023-10-20T03:45:29.950 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42286.log 2023-10-20T03:45:29.950 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43986.log.gz 2023-10-20T03:45:29.950 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39231.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-20T03:45:29.951 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39231.log.gz 2023-10-20T03:45:29.951 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40231.log 2023-10-20T03:45:29.951 INFO:teuthology.orchestra.run.smithi033.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.42286.log.gz 2023-10-20T03:45:29.951 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39431.log 2023-10-20T03:45:29.951 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41295.log 2023-10-20T03:45:29.952 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40231.log.gz 2023-10-20T03:45:29.952 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45473.log 2023-10-20T03:45:29.952 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39431.log.gz 2023-10-20T03:45:29.952 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39667.log 2023-10-20T03:45:29.952 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41295.log.gz 2023-10-20T03:45:29.953 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39658.log 2023-10-20T03:45:29.953 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45473.log.gz 2023-10-20T03:45:29.953 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44693.log 2023-10-20T03:45:29.953 INFO:teuthology.orchestra.run.smithi033.stderr: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39667.log.gz 2023-10-20T03:45:29.953 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-10-20T03:45:29.954 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39010.log 2023-10-20T03:45:29.954 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39658.log.gz 2023-10-20T03:45:29.954 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39251.log 2023-10-20T03:45:29.954 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44693.log.gz 2023-10-20T03:45:29.954 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39010.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2023-10-20T03:45:29.954 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39010.log.gz 2023-10-20T03:45:29.955 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46180.log 2023-10-20T03:45:29.955 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39251.log.gz 2023-10-20T03:45:29.955 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44325.log 2023-10-20T03:45:29.955 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39710.log 2023-10-20T03:45:29.955 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46180.log.gz 2023-10-20T03:45:29.956 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44589.log 2023-10-20T03:45:29.956 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44325.log.gz 2023-10-20T03:45:29.956 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40613.log 2023-10-20T03:45:29.956 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39710.log.gz 2023-10-20T03:45:29.956 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45343.log 2023-10-20T03:45:29.956 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44589.log.gz 2023-10-20T03:45:29.957 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43622.log 2023-10-20T03:45:29.957 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40613.log.gz 2023-10-20T03:45:29.957 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44509.log 2023-10-20T03:45:29.957 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45343.log.gz 2023-10-20T03:45:29.957 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44221.log 2023-10-20T03:45:29.957 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43622.log.gz 2023-10-20T03:45:29.958 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41812.log 2023-10-20T03:45:29.958 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44509.log.gz 2023-10-20T03:45:29.958 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46045.log 2023-10-20T03:45:29.958 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44221.log.gz 2023-10-20T03:45:29.958 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45707.log 2023-10-20T03:45:29.959 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41812.log.gz 2023-10-20T03:45:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42954.log 2023-10-20T03:45:29.959 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46045.log.gz 2023-10-20T03:45:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46363.log 2023-10-20T03:45:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45707.log.gz 2023-10-20T03:45:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35821.log 2023-10-20T03:45:29.960 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42954.log.gz 2023-10-20T03:45:29.960 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41622.log 2023-10-20T03:45:29.960 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46363.log.gz 2023-10-20T03:45:29.960 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40693.log 2023-10-20T03:45:29.960 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35821.log.gz 2023-10-20T03:45:29.961 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45239.log 2023-10-20T03:45:29.961 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41622.log.gz 2023-10-20T03:45:29.961 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42072.log 2023-10-20T03:45:29.961 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40693.log.gz 2023-10-20T03:45:29.961 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43302.log 2023-10-20T03:45:29.962 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45239.log.gz 2023-10-20T03:45:29.962 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45317.log 2023-10-20T03:45:29.962 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42072.log.gz 2023-10-20T03:45:29.962 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39247.log 2023-10-20T03:45:29.962 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43302.log.gz 2023-10-20T03:45:29.962 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41249.log 2023-10-20T03:45:29.963 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45317.log.gz 2023-10-20T03:45:29.963 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39711.log 2023-10-20T03:45:29.963 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39247.log.gz 2023-10-20T03:45:29.963 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42046.log 2023-10-20T03:45:29.963 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41249.log.gz 2023-10-20T03:45:29.964 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39711.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30811.log 2023-10-20T03:45:29.964 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39711.log.gz 2023-10-20T03:45:29.964 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45629.log 2023-10-20T03:45:29.964 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42046.log.gz 2023-10-20T03:45:29.964 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.tmp-client.admin.30811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40328.log 2023-10-20T03:45:29.965 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30811.log.gz 2023-10-20T03:45:29.965 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45629.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-20T03:45:29.965 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45629.log.gz 2023-10-20T03:45:29.965 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39226.log 2023-10-20T03:45:29.965 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40328.log.gz 2023-10-20T03:45:29.966 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41114.log 2023-10-20T03:45:29.966 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39228.log 2023-10-20T03:45:29.966 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39226.log.gz 2023-10-20T03:45:29.966 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45213.log 2023-10-20T03:45:29.966 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41114.log.gz 2023-10-20T03:45:29.967 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40929.log 2023-10-20T03:45:29.967 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.39228.log: /var/log/ceph/ceph-client.admin.45213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42463.log 2023-10-20T03:45:29.967 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39228.log.gz 2023-10-20T03:45:29.967 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45213.log.gz 2023-10-20T03:45:29.967 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40929.log.gz 2023-10-20T03:45:29.967 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42874.log 2023-10-20T03:45:29.968 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41707.log 2023-10-20T03:45:29.968 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42463.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42463.log.gz 2023-10-20T03:45:29.968 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.42874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42874.log.gz 2023-10-20T03:45:29.968 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46019.log 2023-10-20T03:45:29.968 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.41707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40585.log 2023-10-20T03:45:29.969 INFO:teuthology.orchestra.run.smithi033.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.41707.log.gz 2023-10-20T03:45:29.969 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46019.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46019.log.gz -5 2023-10-20T03:45:29.969 INFO:teuthology.orchestra.run.smithi033.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45187.log 2023-10-20T03:45:29.969 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45369.log 2023-10-20T03:45:29.969 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40585.log.gz 2023-10-20T03:45:29.970 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44667.log 2023-10-20T03:45:29.970 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.45187.log: /var/log/ceph/ceph-client.admin.45369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46124.log 2023-10-20T03:45:29.970 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45187.log.gz 2023-10-20T03:45:29.970 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45369.log.gz 2023-10-20T03:45:29.970 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44667.log.gz 2023-10-20T03:45:29.970 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44615.log 2023-10-20T03:45:29.971 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43934.log 2023-10-20T03:45:29.971 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.46124.log: /var/log/ceph/ceph-client.admin.44615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44168.log 2023-10-20T03:45:29.971 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46124.log.gz 2023-10-20T03:45:29.971 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44615.log.gz 2023-10-20T03:45:29.971 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.43934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43934.log.gz 2023-10-20T03:45:29.972 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45733.log 2023-10-20T03:45:29.972 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44797.log 2023-10-20T03:45:29.972 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44168.log: /var/log/ceph/ceph-client.admin.45733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40830.log 2023-10-20T03:45:29.972 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44168.log.gz 2023-10-20T03:45:29.972 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45733.log.gz 2023-10-20T03:45:29.973 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.44797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44797.log.gz 2023-10-20T03:45:29.973 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.40830.log: 24.3% -- replaced with /var/log/ceph/ceph-client.admin.40830.log.gz 2023-10-20T03:45:29.973 INFO:teuthology.orchestra.run.smithi170.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-20T03:45:29.982 INFO:teuthology.orchestra.run.smithi040.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-20T03:45:30.007 INFO:teuthology.orchestra.run.smithi064.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-20T03:45:30.110 INFO:teuthology.orchestra.run.smithi040.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2023-10-20T03:45:30.162 INFO:teuthology.orchestra.run.smithi033.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-20T03:45:30.189 INFO:teuthology.orchestra.run.smithi040.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-20T03:45:30.237 INFO:teuthology.orchestra.run.smithi064.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-20T03:45:30.262 INFO:teuthology.orchestra.run.smithi170.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2023-10-20T03:45:30.266 INFO:teuthology.orchestra.run.smithi170.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-20T03:45:30.269 INFO:teuthology.orchestra.run.smithi064.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-20T03:45:30.279 INFO:teuthology.orchestra.run.smithi033.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2023-10-20T03:45:30.286 INFO:teuthology.orchestra.run.smithi033.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2023-10-20T03:45:30.331 INFO:teuthology.orchestra.run.smithi033.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-20T03:45:30.352 INFO:teuthology.orchestra.run.smithi064.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2023-10-20T03:45:30.370 INFO:teuthology.orchestra.run.smithi170.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2023-10-20T03:45:30.370 INFO:teuthology.orchestra.run.smithi064.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2023-10-20T03:45:30.371 INFO:teuthology.orchestra.run.smithi064.stderr: 2023-10-20T03:45:30.372 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m0.578s 2023-10-20T03:45:30.372 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m2.177s 2023-10-20T03:45:30.372 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.103s 2023-10-20T03:45:30.377 INFO:teuthology.orchestra.run.smithi040.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-20T03:45:30.426 INFO:teuthology.orchestra.run.smithi170.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-20T03:45:30.427 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-10-20T03:45:30.427 INFO:teuthology.orchestra.run.smithi170.stderr:real 0m0.615s 2023-10-20T03:45:30.428 INFO:teuthology.orchestra.run.smithi170.stderr:user 0m2.067s 2023-10-20T03:45:30.428 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m0.085s 2023-10-20T03:45:30.487 INFO:teuthology.orchestra.run.smithi033.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-20T03:45:30.488 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-10-20T03:45:30.488 INFO:teuthology.orchestra.run.smithi033.stderr:real 0m0.722s 2023-10-20T03:45:30.488 INFO:teuthology.orchestra.run.smithi033.stderr:user 0m2.591s 2023-10-20T03:45:30.488 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m0.141s 2023-10-20T03:45:30.543 INFO:teuthology.orchestra.run.smithi040.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2023-10-20T03:45:30.561 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-20T03:45:30.561 INFO:teuthology.orchestra.run.smithi040.stderr:real 0m0.776s 2023-10-20T03:45:30.561 INFO:teuthology.orchestra.run.smithi040.stderr:user 0m2.249s 2023-10-20T03:45:30.561 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m0.077s 2023-10-20T03:45:30.562 INFO:tasks.ceph:Archiving logs... 2023-10-20T03:45:30.562 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/remote/smithi033/log 2023-10-20T03:45:30.563 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-20T03:45:31.119 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/remote/smithi040/log 2023-10-20T03:45:31.120 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-20T03:45:31.415 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/remote/smithi064/log 2023-10-20T03:45:31.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-20T03:45:31.711 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/remote/smithi170/log 2023-10-20T03:45:31.712 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-20T03:45:32.015 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-20T03:45:32.016 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-20T03:45:32.026 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-10-20T03:45:32.027 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-20T03:45:32.027 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-20T03:45:32.032 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-20T03:45:32.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-20T03:45:32.040 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-20T03:45:32.151 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-20T03:45:32.161 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-20T03:45:32.162 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-20T03:45:32.165 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-20T03:45:32.168 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-20T03:45:32.170 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-20T03:45:32.176 INFO:teuthology.orchestra.run.smithi033.stdout: remote refid st t when poll reach delay offset jitter 2023-10-20T03:45:32.176 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================== 2023-10-20T03:45:32.176 INFO:teuthology.orchestra.run.smithi033.stdout:*hv01.front.sepi 44.4.53.2 2 u 54 64 377 0.093 -0.135 1.858 2023-10-20T03:45:32.177 INFO:teuthology.orchestra.run.smithi033.stdout:+hv02.front.sepi 63.231.80.2 3 u 47 64 377 0.096 -0.500 1.241 2023-10-20T03:45:32.177 INFO:teuthology.orchestra.run.smithi033.stdout:+hv03.front.sepi 74.6.168.72 3 u 48 64 377 0.083 -3.528 1.920 2023-10-20T03:45:32.177 INFO:teuthology.orchestra.run.smithi033.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2023-10-20T03:45:32.179 INFO:teuthology.orchestra.run.smithi040.stdout: remote refid st t when poll reach delay offset jitter 2023-10-20T03:45:32.179 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================== 2023-10-20T03:45:32.179 INFO:teuthology.orchestra.run.smithi040.stdout:*hv01.front.sepi 44.4.53.2 2 u 52 64 377 0.067 -1.798 0.499 2023-10-20T03:45:32.179 INFO:teuthology.orchestra.run.smithi040.stdout:+hv02.front.sepi 63.231.80.2 3 u 55 64 377 0.097 -1.324 0.491 2023-10-20T03:45:32.179 INFO:teuthology.orchestra.run.smithi040.stdout:+hv03.front.sepi 74.6.168.72 3 u 53 64 377 0.112 -1.516 0.552 2023-10-20T03:45:32.180 INFO:teuthology.orchestra.run.smithi040.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2023-10-20T03:45:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: remote refid st t when poll reach delay offset jitter 2023-10-20T03:45:32.181 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================== 2023-10-20T03:45:32.181 INFO:teuthology.orchestra.run.smithi064.stdout:*hv01.front.sepi 44.4.53.2 2 u 52 64 377 0.105 -2.388 1.237 2023-10-20T03:45:32.181 INFO:teuthology.orchestra.run.smithi064.stdout:+hv02.front.sepi 63.231.80.2 3 u 43 64 377 0.111 -1.950 1.251 2023-10-20T03:45:32.181 INFO:teuthology.orchestra.run.smithi064.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.103 -1.213 1.322 2023-10-20T03:45:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2023-10-20T03:45:32.183 INFO:teuthology.orchestra.run.smithi170.stdout: remote refid st t when poll reach delay offset jitter 2023-10-20T03:45:32.183 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================== 2023-10-20T03:45:32.184 INFO:teuthology.orchestra.run.smithi170.stdout:*hv01.front.sepi 44.4.53.2 2 u 56 64 377 0.111 -1.164 2.169 2023-10-20T03:45:32.184 INFO:teuthology.orchestra.run.smithi170.stdout:+hv02.front.sepi 63.231.80.2 3 u 48 64 377 0.101 -0.212 2.618 2023-10-20T03:45:32.184 INFO:teuthology.orchestra.run.smithi170.stdout:+hv03.front.sepi 74.6.168.72 3 u 49 64 377 0.100 -3.797 1.982 2023-10-20T03:45:32.184 INFO:teuthology.orchestra.run.smithi170.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2023-10-20T03:45:32.185 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-20T03:45:32.207 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-20T03:45:32.208 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-20T03:45:32.218 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-20T03:45:32.228 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-20T03:45:32.237 INFO:teuthology.task.internal:Duration was 1322.939538 seconds 2023-10-20T03:45:32.238 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-20T03:45:32.248 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-20T03:45:32.249 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-20T03:45:32.251 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-20T03:45:32.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-20T03:45:32.256 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-20T03:45:32.341 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-20T03:45:32.342 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2023-10-20T03:45:32.342 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-20T03:45:32.353 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2023-10-20T03:45:32.354 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-20T03:45:32.363 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2023-10-20T03:45:32.364 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-20T03:45:32.373 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2023-10-20T03:45:32.374 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-20T03:45:32.384 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-20T03:45:32.385 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-20T03:45:32.396 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-20T03:45:32.410 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-20T03:45:32.418 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-20T03:45:32.447 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-20T03:45:32.447 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-20T03:45:32.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-20T03:45:32.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-20T03:45:32.482 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-20T03:45:32.544 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-20T03:45:32.554 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-20T03:45:32.555 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-20T03:45:32.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-20T03:45:32.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-20T03:45:32.594 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-20T03:45:32.608 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-20T03:45:32.618 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-20T03:45:32.621 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-20T03:45:32.626 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-20T03:45:32.630 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2023-10-20T03:45:32.635 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2023-10-20T03:45:32.638 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-20T03:45:32.648 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2023-10-20T03:45:32.661 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2023-10-20T03:45:32.931 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-20T03:45:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:45:32.938 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-20T03:45:32.943 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi040.front.sepia.ceph.com, flagging run as failed 2023-10-20T03:45:32.944 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-20T03:45:32.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:45:32.950 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-20T03:45:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:45:32.978 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-20T03:45:32.989 INFO:teuthology.task.internal:Transferring archived files... 2023-10-20T03:45:32.990 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/remote/smithi033 2023-10-20T03:45:32.990 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-20T03:45:33.010 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/remote/smithi040 2023-10-20T03:45:33.011 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-20T03:45:55.081 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2023-10-20T03:45:55.094 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476/remote/smithi040/coredump/1697772975.35553.core.gz: gzip compressed data, was "1697772975.35553.core", last modified: Fri Oct 20 03:45:33 2023, max compression, original size modulo 2^32 798285824 2023-10-20T03:45:55.095 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2023-10-20T03:45:58.253 INFO:teuthology.task.internal: core looks like: /tmp/tmptdps227c: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-osd -f --cluster ceph -i 9', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/ceph-osd', platform: 'x86_64' 2023-10-20T03:45:58.337 INFO:teuthology.task.internal: dump_program: ceph-osd -f --cluster ceph -i 9 2023-10-20T03:45:58.338 DEBUG:teuthology.orchestra.run.smithi040:> which 'ceph-osd -f --cluster ceph -i 9' 2023-10-20T03:45:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:45:58.347 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 1: "which 'ceph-osd -f --cluster ceph -i 9'" 2023-10-20T03:45:58.348 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-20T03:45:58.360 INFO:teuthology.task.internal:Not uploading archives. 2023-10-20T03:45:58.360 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-20T03:45:58.379 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-20T03:45:58.380 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-20T03:45:58.383 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-20T03:45:58.388 INFO:teuthology.orchestra.run.smithi033.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 20 03:45 /home/ubuntu/cephtest 2023-10-20T03:45:58.388 INFO:teuthology.orchestra.run.smithi033.stdout: 658618 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 20 03:45 /home/ubuntu/cephtest/archive 2023-10-20T03:45:58.388 INFO:teuthology.orchestra.run.smithi033.stdout: 658620 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog 2023-10-20T03:45:58.388 INFO:teuthology.orchestra.run.smithi033.stdout: 658624 4 -rw-rw-rw- 1 ubuntu ubuntu 819 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-10-20T03:45:58.388 INFO:teuthology.orchestra.run.smithi033.stdout: 658621 100 -rw-rw-r-- 1 ubuntu ubuntu 101482 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-20T03:45:58.389 INFO:teuthology.orchestra.run.smithi033.stdout: 663678 88 -rw-rw-rw- 1 ubuntu ubuntu 89194 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-10-20T03:45:58.390 INFO:teuthology.orchestra.run.smithi033.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-20T03:45:58.391 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-20T03:45:58.394 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-20T03:45:58.395 INFO:teuthology.orchestra.run.smithi040.stdout: 655473 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 20 03:45 /home/ubuntu/cephtest 2023-10-20T03:45:58.396 INFO:teuthology.orchestra.run.smithi040.stdout: 658636 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Oct 20 03:23 /home/ubuntu/cephtest/archive 2023-10-20T03:45:58.396 INFO:teuthology.orchestra.run.smithi040.stdout: 658638 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog 2023-10-20T03:45:58.396 INFO:teuthology.orchestra.run.smithi040.stdout: 655471 4 -rw-rw-rw- 1 ubuntu ubuntu 828 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-10-20T03:45:58.396 INFO:teuthology.orchestra.run.smithi040.stdout: 658639 96 -rw-rw-r-- 1 ubuntu ubuntu 96916 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-20T03:45:58.396 INFO:teuthology.orchestra.run.smithi040.stdout: 655432 84 -rw-rw-rw- 1 ubuntu ubuntu 82541 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-10-20T03:45:58.397 INFO:teuthology.orchestra.run.smithi040.stdout: 658637 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 20 03:36 /home/ubuntu/cephtest/archive/coredump 2023-10-20T03:45:58.397 INFO:teuthology.orchestra.run.smithi040.stdout: 663593 72768 -rw------- 1 root root 798285824 Oct 20 03:36 /home/ubuntu/cephtest/archive/coredump/1697772975.35553.core 2023-10-20T03:45:58.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-20T03:45:58.398 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_8a5df55d22d3638066184c241442ff00fdc0ebdd/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-20T03:45:58.400 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-20T03:45:58.411 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-20T03:45:58.424 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-10-20T03:45:58.424 INFO:teuthology.nuke:Checking targets against current locks 2023-10-20T03:45:58.426 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-20T03:45:58.429 INFO:teuthology.orchestra.run.smithi064.stdout: 655471 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 20 03:45 /home/ubuntu/cephtest 2023-10-20T03:45:58.429 INFO:teuthology.orchestra.run.smithi064.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 20 03:45 /home/ubuntu/cephtest/archive 2023-10-20T03:45:58.430 INFO:teuthology.orchestra.run.smithi064.stdout: 655474 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog 2023-10-20T03:45:58.430 INFO:teuthology.orchestra.run.smithi064.stdout: 658614 4 -rw-rw-rw- 1 ubuntu ubuntu 838 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-10-20T03:45:58.430 INFO:teuthology.orchestra.run.smithi064.stdout: 658613 96 -rw-rw-r-- 1 ubuntu ubuntu 96443 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-20T03:45:58.430 INFO:teuthology.orchestra.run.smithi064.stdout: 663667 84 -rw-rw-rw- 1 ubuntu ubuntu 82173 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-10-20T03:45:58.431 INFO:teuthology.orchestra.run.smithi170.stdout: 655475 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 20 03:45 /home/ubuntu/cephtest 2023-10-20T03:45:58.431 INFO:teuthology.orchestra.run.smithi170.stdout: 658615 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 20 03:45 /home/ubuntu/cephtest/archive 2023-10-20T03:45:58.431 INFO:teuthology.orchestra.run.smithi170.stdout: 658617 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog 2023-10-20T03:45:58.431 INFO:teuthology.orchestra.run.smithi170.stdout: 658621 4 -rw-rw-rw- 1 ubuntu ubuntu 796 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-10-20T03:45:58.431 INFO:teuthology.orchestra.run.smithi170.stdout: 658618 96 -rw-rw-r-- 1 ubuntu ubuntu 96656 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-20T03:45:58.432 INFO:teuthology.orchestra.run.smithi170.stdout: 663672 84 -rw-rw-rw- 1 ubuntu ubuntu 82437 Oct 20 03:45 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-10-20T03:45:58.433 INFO:teuthology.orchestra.run.smithi064.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-20T03:45:58.433 INFO:teuthology.orchestra.run.smithi170.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-20T03:45:58.457 DEBUG:teuthology.nuke:shortname: smithi033 2023-10-20T03:45:58.458 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-20T03:45:58.481 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-20 03:14:22.762626', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKVOLgduYY8Px3G6GDW8+lkODtfqJDRiQdtvy3CJdcBka0VxAiKn06fsijKrvDtLUnWqN7FSFQQxQL+f/uE7L0A='} 2023-10-20T03:45:58.485 DEBUG:teuthology.nuke:shortname: smithi040 2023-10-20T03:45:58.486 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-20T03:45:58.508 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-20 03:14:22.766373', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPl2EW4umy7LmnRQKJV/eZwxZaWNyUCOhFdnsV1xnxguPQ3cGwHK2gDCmNjhoifsNP69ssqA/e2XaUh7m5UejZo='} 2023-10-20T03:45:58.513 DEBUG:teuthology.nuke:shortname: smithi064 2023-10-20T03:45:58.513 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-20T03:45:58.541 INFO:teuthology.orchestra.console.smithi033:Power off 2023-10-20T03:45:58.542 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-20T03:45:58.553 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-20 03:14:22.764678', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEthpyLSta/xLwOp5moRS9tdSUTC7b2/1Ub3fauZiI4GC4QFVwvwI9RehZegOcyuYvXLyZr+5XHJP/D2yxtj0Lk='} 2023-10-20T03:45:58.556 DEBUG:teuthology.nuke:shortname: smithi170 2023-10-20T03:45:58.556 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-20T03:45:58.565 DEBUG:teuthology.orchestra.console.smithi033:power off output: Chassis Power Control: Down/Off 2023-10-20T03:45:58.565 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:45:58.589 INFO:teuthology.orchestra.console.smithi040:Power off 2023-10-20T03:45:58.589 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-20T03:45:58.600 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-19_16:23:25-rados-wip-yuri10-testing-2023-10-11-0812-distro-default-smithi/7432476', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-20 03:14:22.768050', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCiptNh76uIuJqcjiE0raXNpFMJh2P8PY0fynAbDKQwiRO6c+CQE0wHU+LuDdd/46qgoWu9/nM+WCTFcmVQmPvQ='} 2023-10-20T03:45:58.605 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-10-20T03:45:58.613 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2023-10-20T03:45:58.614 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:45:58.636 INFO:teuthology.orchestra.console.smithi064:Power off 2023-10-20T03:45:58.637 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-20T03:45:58.647 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-10-20T03:45:58.661 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2023-10-20T03:45:58.661 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:45:58.680 INFO:teuthology.orchestra.console.smithi170:Power off 2023-10-20T03:45:58.680 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-20T03:45:58.690 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2023-10-20T03:45:58.704 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2023-10-20T03:45:58.704 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:45:58.728 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-10-20T03:46:02.606 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:46:02.648 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:46:02.692 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:46:02.717 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-10-20T03:46:02.729 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:46:02.759 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-10-20T03:46:02.802 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2023-10-20T03:46:02.839 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-10-20T03:46:06.722 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:46:06.760 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:46:06.803 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:46:06.833 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-10-20T03:46:06.840 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:46:06.871 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-10-20T03:46:06.914 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2023-10-20T03:46:06.950 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2023-10-20T03:46:07.051 INFO:teuthology.orchestra.console.smithi170:Power off completed 2023-10-20T03:46:10.838 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:46:10.873 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:46:10.915 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-20T03:46:10.949 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is off 2023-10-20T03:46:10.984 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2023-10-20T03:46:11.026 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2023-10-20T03:46:11.051 INFO:teuthology.orchestra.console.smithi033:Power off completed 2023-10-20T03:46:11.085 INFO:teuthology.orchestra.console.smithi040:Power off completed 2023-10-20T03:46:11.127 INFO:teuthology.orchestra.console.smithi064:Power off completed 2023-10-20T03:46:11.232 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 1322.9395380020142 failure_reason: 'Command failed on smithi033 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json''' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2023-10-20T03:46:11.233 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-20T03:46:11.376 INFO:teuthology.run:FAIL